Skip to content
Snippets Groups Projects
Commit 27c420d1 authored by CI-bot's avatar CI-bot Committed by renku 0.10.4
Browse files

renku rerun data/covidtracking/states-metadata.json data/covidtracking/states-daily.json

parent 35f45688
No related branches found
No related tags found
1 merge request!222Automatic update - auto-update_2020-06-18_07-42
class: Workflow
cwlVersion: v1.0
hints: []
inputs:
input_1:
default: states-daily.json
streamable: false
type: string
input_2:
default: states-metadata.json
streamable: false
type: string
input_3:
default: out_folder
streamable: false
type: string
input_4:
default: data/covidtracking
streamable: false
type: string
input_5:
default:
class: File
path: ../../notebooks/process/download-covidtracking-data.ipynb
streamable: false
type: File
input_6:
default: runs/download-covidtracking-data.runs.ipynb
streamable: false
type: string
outputs:
output_0:
outputSource: step_3/output_0
streamable: false
type: File
output_2:
outputSource: step_3/output_1
streamable: false
type: Directory
requirements: []
steps:
step_1:
in:
filename: input_1
input_directory: step_3/output_1
out:
- output_file
run:
arguments: []
baseCommand:
- 'true'
class: CommandLineTool
cwlVersion: v1.0
hints: []
inputs:
filename:
default: states-daily.json
streamable: false
type: string
input_directory:
streamable: false
type: Directory
outputs:
output_file:
outputBinding:
glob: $(inputs.filename)
streamable: false
type: File
permanentFailCodes: []
requirements:
- &id001
class: InlineJavascriptRequirement
- &id002
class: InitialWorkDirRequirement
listing: $(inputs.input_directory.listing)
successCodes: []
temporaryFailCodes: []
step_2:
in:
filename: input_2
input_directory: step_3/output_1
out:
- output_file
run:
arguments: []
baseCommand:
- 'true'
class: CommandLineTool
cwlVersion: v1.0
hints: []
inputs:
filename:
default: states-metadata.json
streamable: false
type: string
input_directory:
streamable: false
type: Directory
outputs:
output_file:
outputBinding:
glob: $(inputs.filename)
streamable: false
type: File
permanentFailCodes: []
requirements:
- *id001
- *id002
successCodes: []
temporaryFailCodes: []
step_3:
in:
input_1: input_3
input_2: input_4
input_3: input_5
input_4: input_6
out:
- output_1
- output_0
run: a17d560c41a54f5aa307ce5f3c5effe5_papermill.cwl
source diff could not be displayed: it is stored in LFS. Options to address this: view the blob.
source diff could not be displayed: it is stored in LFS. Options to address this: view the blob.
%% Cell type:code id: tags:
``` python
import requests
import os
import pandas as pd
```
%% Cell type:code id: tags:parameters
``` python
out_folder = "../data/covidtracking/"
PAPERMILL_OUTPUT_PATH = None
```
%% Cell type:code id: tags:injected-parameters
``` python
# Parameters
PAPERMILL_INPUT_PATH = "/tmp/um7bcmv4/notebooks/process/download-covidtracking-data.ipynb"
PAPERMILL_INPUT_PATH = "/tmp/y5_1jzrh/notebooks/process/download-covidtracking-data.ipynb"
PAPERMILL_OUTPUT_PATH = "runs/download-covidtracking-data.runs.ipynb"
out_folder = "data/covidtracking"
```
%% Cell type:markdown id: tags:
# Download state metadata
Download a dataset of URLs for data for each US state and several territories. See [Google Doc](https://docs.google.com/spreadsheets/d/18oVRrHj3c183mHmq3m89_163yuYltLNlOmPerQ18E8w/htmlview?sle=true).
%% Cell type:code id: tags:
``` python
url = 'http://covidtracking.com/api/states/info'
r = requests.get(url, allow_redirects=True)
states_metadata_json = r.content
```
%% Cell type:code id: tags:
``` python
# save the result
if PAPERMILL_OUTPUT_PATH:
out_path = os.path.join(out_folder, 'states-metadata.json')
with open(out_path, 'wb') as f:
f.write(states_metadata_json)
```
%% Cell type:code id: tags:
``` python
metadata_df = pd.read_json(states_metadata_json)
print(len(metadata_df), "states and territories have metadata")
metadata_df.head(2)
```
%% Output
56 states and territories have metadata
state notes \
0 AK Total tests are taken from the annotations on ...
1 AL Negatives = (Totals - Positives) \nPositives o...
covid19Site \
0 http://dhss.alaska.gov/dph/Epi/id/Pages/COVID-...
1 https://alpublichealth.maps.arcgis.com/apps/op...
covid19SiteSecondary covid19SiteTertiary \
0 http://dhss.alaska.gov/dph/Epi/id/Pages/COVID-...
1 https://dph1.adph.state.al.us/covid-19/
twitter covid19SiteOld \
0 @Alaska_DHSS http://dhss.alaska.gov/dph/Epi/id/Pages/COVID-...
1 @alpublichealth http://www.alabamapublichealth.gov/infectiousd...
name fips pui pum
0 Alaska 2 False
1 Alabama 1 False
%% Cell type:markdown id: tags:
# Download daily state data
%% Cell type:code id: tags:
``` python
url = 'https://covidtracking.com/api/states/daily'
r = requests.get(url, allow_redirects=True)
states_daily_json = r.content
```
%% Cell type:code id: tags:
``` python
# save the result
if PAPERMILL_OUTPUT_PATH:
out_path = os.path.join(out_folder, 'states-daily.json')
with open(out_path, 'wb') as f:
f.write(states_daily_json)
```
%% Cell type:code id: tags:
``` python
data_df = pd.read_json(states_daily_json)
print(len(data_df), "data points")
data_df.head(2)
```
%% Output
5785 data points
5841 data points
date state positive negative pending hospitalizedCurrently \
0 20200616 AK 676.0 75539.0 NaN 22.0
1 20200616 AL 26912.0 278512.0 NaN 678.0
0 20200617 AK 696.0 77013.0 NaN 23.0
1 20200617 AL 27312.0 283411.0 NaN 677.0
hospitalizedCumulative inIcuCurrently inIcuCumulative \
0 NaN NaN NaN
1 2315.0 NaN 682.0
1 2352.0 NaN 687.0
onVentilatorCurrently ... posNeg deathIncrease hospitalizedIncrease \
0 2.0 ... 76215 0 0
1 NaN ... 305424 11 56
0 0.0 ... 77709 0 0
1 NaN ... 310723 5 37
hash commercialScore \
0 9ce63702a549318053a366a0ef2d97680c2e6bad 0
1 e76cb0a1b1984ba8a95f80bd42da175e42add663 0
0 cb5c6333f1f29241387021d8d30f0c2efad0444c 0
1 bd6edf2bb5b10f200f5e0636f87da980ae88a0cc 0
negativeRegularScore negativeScore positiveScore score grade
0 0 0 0 0
1 0 0 0 0
[2 rows x 39 columns]
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment