Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
D
democrasci_preprocWP1
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Container Registry
Model registry
Operate
Environments
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
Luis Salamanca
democrasci_preprocWP1
Commits
09ce8031
Commit
09ce8031
authored
6 years ago
by
Lili Gasser
Browse files
Options
Downloads
Patches
Plain Diff
add french training data
parent
e17e664e
No related branches found
Branches containing commit
No related tags found
No related merge requests found
Pipeline
#1938
passed
6 years ago
Stage: build
Changes
3
Pipelines
1
Expand all
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
.gitignore
+1
-0
1 addition, 0 deletions
.gitignore
notebooks/NER_read-training-data.ipynb
+5149
-52
5149 additions, 52 deletions
notebooks/NER_read-training-data.ipynb
src/python/utils_ner.py
+11
-4
11 additions, 4 deletions
src/python/utils_ner.py
with
5161 additions
and
56 deletions
.gitignore
+
1
−
0
View file @
09ce8031
...
...
@@ -281,3 +281,4 @@ Pipfile.lock
# NER training files
data/train_NER/1[0-9][0-9][0-9]_20[0-9][0-9][0-9][0-9][0-9][0-9].txt
data/train_NER/1[0-9][0-9][0-9]_20[0-9][0-9][0-9][0-9][0-9][0-9]_french.txt
This diff is collapsed.
Click to expand it.
notebooks/NER_read-training-data.ipynb
+
5149
−
52
View file @
09ce8031
This diff is collapsed.
Click to expand it.
src/python/utils_ner.py
+
11
−
4
View file @
09ce8031
...
...
@@ -6,7 +6,8 @@ colors = {'ORG': '#73c6b6', 'LOC': '#bb8fce', 'PER': '#e59866', 'MISC': '#a6acaf
options
=
{
'
ents
'
:
[
'
ORG
'
,
'
LOC
'
,
'
PER
'
,
'
MISC
'
],
'
colors
'
:
colors
}
colors_sner
=
{
'
ORGANIZATION
'
:
'
#73c6b6
'
,
'
LOCATION
'
:
'
#bb8fce
'
,
'
PERSON
'
:
'
#e59866
'
,
'
MISC
'
:
'
#a6acaf
'
}
options_sner
=
{
'
ents
'
:
[
'
ORGANIZATION
'
,
'
LOCATION
'
,
'
PERSON
'
,
'
MISC
'
],
'
colors
'
:
colors_sner
}
colors_sner_french
=
{
'
I-ORG
'
:
'
#73c6b6
'
,
'
I-LIEU
'
:
'
#bb8fce
'
,
'
I-PERS
'
:
'
#e59866
'
}
options_sner_french
=
{
'
ents
'
:
[
'
I-ORG
'
,
'
I-LOC
'
,
'
I-PERS
'
],
'
colors
'
:
colors_sner_french
}
...
...
@@ -75,9 +76,13 @@ def read_from_txt(filename):
pass
return
alldicts
def
render_dict
(
alldicts
):
def
render_dict
(
alldicts
,
language
):
if
language
==
'
german
'
:
options
=
options_sner
if
language
==
'
french
'
:
options
=
options_sner_french
for
sent_key
,
sent_dict
in
alldicts
.
items
():
displacy
.
render
(
sent_dict
,
style
=
'
ent
'
,
jupyter
=
True
,
manual
=
True
,
options
=
options
_sner
)
displacy
.
render
(
sent_dict
,
style
=
'
ent
'
,
jupyter
=
True
,
manual
=
True
,
options
=
options
)
def
transform_to_training_format
(
alldicts
):
...
...
@@ -113,4 +118,6 @@ def transform_to_training_format(alldicts):
return
train_data
def
get_language
(
filepath
):
language
=
'
french
'
if
filepath
.
endswith
(
'
_french.txt
'
)
else
'
german
'
return
language
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment