Skip to content
Snippets Groups Projects
Commit cef673ed authored by CR (covid cron)'s avatar CR (covid cron) Committed by renku 0.10.4.dev13
Browse files

renku rerun data/covidtracking/states-metadata.json data/covidtracking/states-daily.json

parent 8ea3c956
No related branches found
No related tags found
1 merge request!151Update 2020 05 04 07 20
Hide whitespace changes
Inline Side-by-side
.renku/workflow/eafea58b25074956a49ec30e7eacf69c.cwl 0 → 100644
+ 120
− 0
View file @ cef673ed
class: Workflow
cwlVersion: v1.0
hints: []
inputs:
input_1:
default: states-daily.json
streamable: false
type: string
input_2:
default: out_folder
streamable: false
type: string
input_3:
default: data/covidtracking
streamable: false
type: string
input_4:
default:
class: File
path: ../../notebooks/process/download-covidtracking-data.ipynb
streamable: false
type: File
input_5:
default: runs/download-covidtracking-data.runs.ipynb
streamable: false
type: string
input_6:
default: states-metadata.json
streamable: false
type: string
outputs:
output_0:
outputSource: step_2/output_1
streamable: false
type: Directory
output_3:
outputSource: step_2/output_0
streamable: false
type: File
requirements: []
steps:
step_1:
in:
filename: input_1
input_directory: step_2/output_1
out:
- output_file
run:
arguments: []
baseCommand:
- 'true'
class: CommandLineTool
cwlVersion: v1.0
hints: []
inputs:
filename:
default: states-daily.json
streamable: false
type: string
input_directory:
streamable: false
type: Directory
outputs:
output_file:
outputBinding:
glob: $(inputs.filename)
streamable: false
type: File
permanentFailCodes: []
requirements:
- &id001
class: InlineJavascriptRequirement
- &id002
class: InitialWorkDirRequirement
listing: $(inputs.input_directory.listing)
successCodes: []
temporaryFailCodes: []
step_2:
in:
input_1: input_2
input_2: input_3
input_3: input_4
input_4: input_5
out:
- output_0
- output_1
run: a17d560c41a54f5aa307ce5f3c5effe5_papermill.cwl
step_3:
in:
filename: input_6
input_directory: step_2/output_1
out:
- output_file
run:
arguments: []
baseCommand:
- 'true'
class: CommandLineTool
cwlVersion: v1.0
hints: []
inputs:
filename:
default: states-metadata.json
streamable: false
type: string
input_directory:
streamable: false
type: Directory
outputs:
output_file:
outputBinding:
glob: $(inputs.filename)
streamable: false
type: File
permanentFailCodes: []
requirements:
- *id001
- *id002
successCodes: []
temporaryFailCodes: []
data/covidtracking/states-daily.json LFS
+ 2
− 2
View file @ cef673ed
source diff could not be displayed: it is stored in LFS. Options to address this: view the blob.
data/covidtracking/states-metadata.json LFS
+ 2
− 2
View file @ cef673ed
source diff could not be displayed: it is stored in LFS. Options to address this: view the blob.
runs/download-covidtracking-data.runs.ipynb
+ 75
− 75
View file @ cef673ed
%% Cell type:code id: tags:
``` python
import requests
import os
import pandas as pd
```
%% Cell type:code id: tags:parameters
``` python
out_folder = "../data/covidtracking/"
PAPERMILL_OUTPUT_PATH = None
```
%% Cell type:code id: tags:injected-parameters
``` python
# Parameters
PAPERMILL_INPUT_PATH = "/tmp/3jpq_9ab/notebooks/process/download-covidtracking-data.ipynb"
PAPERMILL_INPUT_PATH = "/tmp/096zx1nj/notebooks/process/download-covidtracking-data.ipynb"
PAPERMILL_OUTPUT_PATH = "runs/download-covidtracking-data.runs.ipynb"
out_folder = "data/covidtracking"
```
%% Cell type:markdown id: tags:
# Download state metadata
Download a dataset of URLs for data for each US state and several territories. See [Google Doc](https://docs.google.com/spreadsheets/d/18oVRrHj3c183mHmq3m89_163yuYltLNlOmPerQ18E8w/htmlview?sle=true).
%% Cell type:code id: tags:
``` python
url = 'http://covidtracking.com/api/states/info'
r = requests.get(url, allow_redirects=True)
states_metadata_json = r.content
```
%% Cell type:code id: tags:
``` python
# save the result
if PAPERMILL_OUTPUT_PATH:
out_path = os.path.join(out_folder, 'states-metadata.json')
with open(out_path, 'wb') as f:
f.write(states_metadata_json)
```
%% Cell type:code id: tags:
``` python
metadata_df = pd.read_json(states_metadata_json)
print(len(metadata_df), "states and territories have metadata")
metadata_df.head(2)
```
%% Output
56 states and territories have metadata
state covid19SiteOld \
0 AK http://dhss.alaska.gov/dph/Epi/id/Pages/COVID-...
1 AL http://www.alabamapublichealth.gov/infectiousd...
covid19Site \
0 http://dhss.alaska.gov/dph/Epi/id/Pages/COVID-...
1 https://alpublichealth.maps.arcgis.com/apps/op...
covid19SiteSecondary twitter \
0 http://dhss.alaska.gov/dph/Epi/id/Pages/COVID-... @Alaska_DHSS
1 None @alpublichealth
1 https://dph1.adph.state.al.us/covid-19/ @alpublichealth
pui pum notes fips \
0 All data False Total tests are taken from the annotations on ... 2
1 No data False Negatives = (Totals - Positives) \nPositives o... 1
name
0 Alaska
1 Alabama
%% Cell type:markdown id: tags:
# Download daily state data
%% Cell type:code id: tags:
``` python
url = 'https://covidtracking.com/api/states/daily'
r = requests.get(url, allow_redirects=True)
states_daily_json = r.content
```
%% Cell type:code id: tags:
``` python
# save the result
if PAPERMILL_OUTPUT_PATH:
out_path = os.path.join(out_folder, 'states-daily.json')
with open(out_path, 'wb') as f:
f.write(states_daily_json)
```
%% Cell type:code id: tags:
``` python
data_df = pd.read_json(states_daily_json)
print(len(data_df), "data points")
data_df.head(2)
```
%% Output
3153 data points
3321 data points
date state positive negative pending hospitalizedCurrently \
0 20200430 AK 355.0 18764.0 NaN 19.0
1 20200430 AL 7019.0 80177.0 NaN NaN
0 20200503 AK 368.0 21210.0 NaN 12.0
1 20200503 AL 7725.0 84775.0 NaN NaN
hospitalizedCumulative inIcuCurrently inIcuCumulative \
0 NaN NaN NaN
1 978.0 NaN 335.0
1 1035.0 NaN 403.0
onVentilatorCurrently ... hospitalized total totalTestResults \
0 NaN ... NaN 19119.0 19119.0
1 NaN ... 978.0 87196.0 87196.0
0 NaN ... NaN 21578.0 21578.0
1 NaN ... 1035.0 92500.0 92500.0
posNeg fips deathIncrease hospitalizedIncrease negativeIncrease \
0 19119.0 2 0.0 0.0 0.0
1 87196.0 1 24.0 33.0 6570.0
0 21578.0 2 0.0 0.0 176.0
1 92500.0 1 2.0 12.0 0.0
positiveIncrease totalTestResultsIncrease
0 0.0 0.0
1 177.0 6747.0
0 3.0 179.0
1 291.0 291.0
[2 rows x 25 columns]
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment

This GitLab is a part of renkulab.io