Skip to content
Snippets Groups Projects
Commit c41914b8 authored by CR (covid cron)'s avatar CR (covid cron) Committed by renku 0.10.0
Browse files

renku rerun data/covidtracking/states-metadata.json data/covidtracking/states-daily.json

parent 003e2191
No related branches found
No related tags found
No related merge requests found
class: Workflow
cwlVersion: v1.0
hints: []
inputs:
input_1:
default: out_folder
streamable: false
type: string
input_2:
default: data/covidtracking
streamable: false
type: string
input_3:
default:
class: File
path: ../../notebooks/process/download-covidtracking-data.ipynb
streamable: false
type: File
input_4:
default: runs/download-covidtracking-data.runs.ipynb
streamable: false
type: string
input_5:
default: states-metadata.json
streamable: false
type: string
input_6:
default: states-daily.json
streamable: false
type: string
outputs:
output_1:
outputSource: step_1/output_1
streamable: false
type: Directory
output_2:
outputSource: step_1/output_0
streamable: false
type: File
requirements: []
steps:
step_1:
in:
input_1: input_1
input_2: input_2
input_3: input_3
input_4: input_4
out:
- output_1
- output_0
run: a17d560c41a54f5aa307ce5f3c5effe5_papermill.cwl
step_2:
in:
filename: input_5
input_directory: step_1/output_1
out:
- output_file
run:
arguments: []
baseCommand:
- 'true'
class: CommandLineTool
cwlVersion: v1.0
hints: []
inputs:
filename:
default: states-metadata.json
streamable: false
type: string
input_directory:
streamable: false
type: Directory
outputs:
output_file:
outputBinding:
glob: $(inputs.filename)
streamable: false
type: File
permanentFailCodes: []
requirements:
- &id001
class: InlineJavascriptRequirement
- &id002
class: InitialWorkDirRequirement
listing: $(inputs.input_directory.listing)
successCodes: []
temporaryFailCodes: []
step_3:
in:
filename: input_6
input_directory: step_1/output_1
out:
- output_file
run:
arguments: []
baseCommand:
- 'true'
class: CommandLineTool
cwlVersion: v1.0
hints: []
inputs:
filename:
default: states-daily.json
streamable: false
type: string
input_directory:
streamable: false
type: Directory
outputs:
output_file:
outputBinding:
glob: $(inputs.filename)
streamable: false
type: File
permanentFailCodes: []
requirements:
- *id001
- *id002
successCodes: []
temporaryFailCodes: []
source diff could not be displayed: it is stored in LFS. Options to address this: view the blob.
source diff could not be displayed: it is stored in LFS. Options to address this: view the blob.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
import requests import requests
import os import os
import pandas as pd import pandas as pd
``` ```
%% Cell type:code id: tags:parameters %% Cell type:code id: tags:parameters
``` python ``` python
out_folder = "../data/covidtracking/" out_folder = "../data/covidtracking/"
PAPERMILL_OUTPUT_PATH = None PAPERMILL_OUTPUT_PATH = None
``` ```
%% Cell type:code id: tags:injected-parameters %% Cell type:code id: tags:injected-parameters
``` python ``` python
# Parameters # Parameters
PAPERMILL_INPUT_PATH = "/tmp/cpvikc_3/notebooks/process/download-covidtracking-data.ipynb" PAPERMILL_INPUT_PATH = "/tmp/lrv5vlu8/notebooks/process/download-covidtracking-data.ipynb"
PAPERMILL_OUTPUT_PATH = "runs/download-covidtracking-data.runs.ipynb" PAPERMILL_OUTPUT_PATH = "runs/download-covidtracking-data.runs.ipynb"
out_folder = "data/covidtracking" out_folder = "data/covidtracking"
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# Download state metadata # Download state metadata
Download a dataset of URLs for data for each US state and several territories. See [Google Doc](https://docs.google.com/spreadsheets/d/18oVRrHj3c183mHmq3m89_163yuYltLNlOmPerQ18E8w/htmlview?sle=true). Download a dataset of URLs for data for each US state and several territories. See [Google Doc](https://docs.google.com/spreadsheets/d/18oVRrHj3c183mHmq3m89_163yuYltLNlOmPerQ18E8w/htmlview?sle=true).
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
url = 'http://covidtracking.com/api/states/info' url = 'http://covidtracking.com/api/states/info'
r = requests.get(url, allow_redirects=True) r = requests.get(url, allow_redirects=True)
states_metadata_json = r.content states_metadata_json = r.content
``` ```
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# save the result # save the result
if PAPERMILL_OUTPUT_PATH: if PAPERMILL_OUTPUT_PATH:
out_path = os.path.join(out_folder, 'states-metadata.json') out_path = os.path.join(out_folder, 'states-metadata.json')
with open(out_path, 'wb') as f: with open(out_path, 'wb') as f:
f.write(states_metadata_json) f.write(states_metadata_json)
``` ```
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
metadata_df = pd.read_json(states_metadata_json) metadata_df = pd.read_json(states_metadata_json)
print(len(metadata_df), "states and territories have metadata") print(len(metadata_df), "states and territories have metadata")
metadata_df.head(2) metadata_df.head(2)
``` ```
%% Output %% Output
56 states and territories have metadata 56 states and territories have metadata
state covid19SiteOld \ state covid19SiteOld \
0 AK http://dhss.alaska.gov/dph/Epi/id/Pages/COVID-... 0 AK http://dhss.alaska.gov/dph/Epi/id/Pages/COVID-...
1 AL http://www.alabamapublichealth.gov/infectiousd... 1 AL http://www.alabamapublichealth.gov/infectiousd...
covid19Site \ covid19Site \
0 http://dhss.alaska.gov/dph/Epi/id/Pages/COVID-... 0 http://dhss.alaska.gov/dph/Epi/id/Pages/COVID-...
1 https://alpublichealth.maps.arcgis.com/apps/op... 1 https://alpublichealth.maps.arcgis.com/apps/op...
covid19SiteSecondary twitter \ covid19SiteSecondary twitter \
0 http://dhss.alaska.gov/dph/Epi/id/Pages/COVID-... @Alaska_DHSS 0 http://dhss.alaska.gov/dph/Epi/id/Pages/COVID-... @Alaska_DHSS
1 None @alpublichealth 1 None @alpublichealth
pui pum notes fips \ pui pum notes fips \
0 All data False We count the reported number as "persons teste... 2 0 All data False Total tests are taken from the annotations on ... 2
1 No data False Negatives = Totals - Positives. Positives seem... 1 1 No data False Negatives = Totals - Positives. Positives seem... 1
name name
0 Alaska 0 Alaska
1 Alabama 1 Alabama
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# Download daily state data # Download daily state data
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
url = 'https://covidtracking.com/api/states/daily' url = 'https://covidtracking.com/api/states/daily'
r = requests.get(url, allow_redirects=True) r = requests.get(url, allow_redirects=True)
states_daily_json = r.content states_daily_json = r.content
``` ```
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
# save the result # save the result
if PAPERMILL_OUTPUT_PATH: if PAPERMILL_OUTPUT_PATH:
out_path = os.path.join(out_folder, 'states-daily.json') out_path = os.path.join(out_folder, 'states-daily.json')
with open(out_path, 'wb') as f: with open(out_path, 'wb') as f:
f.write(states_daily_json) f.write(states_daily_json)
``` ```
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
data_df = pd.read_json(states_daily_json) data_df = pd.read_json(states_daily_json)
print(len(data_df), "data points") print(len(data_df), "data points")
data_df.head(2) data_df.head(2)
``` ```
%% Output %% Output
1149 data points 1205 data points
date state positive negative pending hospitalized death total \ date state positive negative pending hospitalized death total \
0 20200326 AK 59.0 1801.0 NaN 3.0 1.0 1860 0 20200327 AK 69.0 2319.0 13.0 3.0 1.0 2401
1 20200326 AL 506.0 3593.0 NaN NaN 1.0 4099 1 20200327 AL 587.0 4184.0 NaN NaN 3.0 4771
dateChecked totalTestResults deathIncrease \ hash dateChecked \
0 2020-03-26T20:00:00Z 1860 0.0 0 588f633d59494e4d58466fccfa9628c98568396b 2020-03-27T20:00:00Z
1 2020-03-26T20:00:00Z 4099 1.0 1 c3883f10c8760fdd9d196c5e2483205434bc4e0f 2020-03-27T20:00:00Z
hospitalizedIncrease negativeIncrease positiveIncrease \ totalTestResults fips deathIncrease hospitalizedIncrease \
0 2.0 152.0 17.0 0 2388 2 0.0 0.0
1 0.0 1064.0 223.0 1 4771 1 2.0 0.0
totalTestResultsIncrease negativeIncrease positiveIncrease totalTestResultsIncrease
0 169.0 0 518.0 10.0 528.0
1 1287.0 1 591.0 81.0 672.0
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment