Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jsl lib docs #152

Merged
merged 8 commits into from
Oct 6, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update h2 title
  • Loading branch information
agsfer committed Oct 5, 2022
commit 84a7e4c065e28e81ea646786a6458598cfecb624
2 changes: 1 addition & 1 deletion docs/_includes/scripts/lib/toc.js
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@
$root.append($tocUl);
$headings.each(function() {
var $this = $(this);
$tocUl.append($('<li style="white-space: normal !important;" ></li>').addClass('toc-' + $this.prop('tagName').toLowerCase())
$tocUl.append($('<li style="white-space: normal !important;" ></li>').addClass('toc-' + $this.prop('tagName').toLowerCase() + ' ' + $this.prop('className'))
.css("white-space", "normal !important")
.append($('<a></a>').text($this.text()).attr('href', '#' + $this.prop('id'))));
});
Expand Down
15 changes: 15 additions & 0 deletions docs/_sass/custom.scss
Original file line number Diff line number Diff line change
Expand Up @@ -344,6 +344,7 @@ header {
li {
margin: 0 0 10px;
white-space: normal;
overflow: visible;
&.toc-h3 {
margin-bottom: 10px;
position: relative;
Expand All @@ -359,6 +360,7 @@ header {
font-size: 15px;
line-height: 20px;
margin: 0;
position: relative;
font-weight: normal;
color: $color-darkblue;
border: none;
Expand All @@ -370,6 +372,19 @@ header {
color: $color-blue;
font-weight: bold;
}
&.h2-select a {
font-size: 16px;
&:before {
content: '';
position: absolute;
width: 100%;
height: 1px;
left: 0;
bottom: -3px;
background: #536b76;
display: block;
}
}
}
}
}
Expand Down
4 changes: 1 addition & 3 deletions docs/en/NLU_under_the_hood.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,7 @@ permalink: /docs/en/under_the_hood
modify_date: "2019-05-16"
---

<div class="main-docs" markdown="1">

<div class="h3-box" markdown="1">
<div class="main-docs" markdown="1"><div class="h3-box" markdown="1">

This page acts as reference on the internal working and implementation of NLU.
It acts as a reference for internal development and open source contributers.
Expand Down
4 changes: 1 addition & 3 deletions docs/en/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,7 @@ key: docs-concepts
modify_date: "2020-05-08"
---

<div class="main-docs" markdown="1">

<div class="h3-box" markdown="1">
<div class="main-docs" markdown="1"><div class="h3-box" markdown="1">

The NLU library provides 2 simple methods with which most NLU tasks can be solved while achieving state of the art results.
The **load** and **predict** method.
Expand Down
43 changes: 18 additions & 25 deletions docs/en/examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,7 @@ permalink: /docs/en/examples
modify_date: "2019-05-16"
---

<div class="main-docs" markdown="1">

<div class="h3-box" markdown="1">
<div class="main-docs" markdown="1"><div class="h3-box" markdown="1">

## Usage examples of NLU.load()
The following examples demonstrate how to use nlu's load api accompanied by the outputs generated by it.
Expand Down Expand Up @@ -1043,7 +1041,7 @@ nlu.load('sentence_detector').predict('NLU can detect things. Like beginning and
|Like beginning and endings of sentences. | [[0.4970400035381317, -0.013454999774694443, 0...]| [NNP, MD, VB, NNS, ., IN, VBG, CC, NNS, IN, NN...]| [O, O, O, O, O, B-sent, O, O, O, O, O, O, B-se...] |
|It can also do much more! | [[0.4970400035381317, -0.013454999774694443, 0...]| [NNP, MD, VB, NNS, ., IN, VBG, CC, NNS, IN, NN...]| [O, O, O, O, O, B-sent, O, O, O, O, O, O, B-se...] |

</div></div></div></div>
</div></div></div><div class="h3-box" markdown="1">



Expand All @@ -1064,6 +1062,8 @@ df
|------|-------------|
| `<!DOCTYPE html> <html> <head> <title>Example</title> </head> <body> <p>This is an example of a simple HTML page with one paragraph.</p> </body> </html>` |Example This is an example of a simple HTML page with one paragraph.|

</div><div class="h3-box" markdown="1">

## Word Segmenter
[Word Segmenter Example](https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/component_examples/multilingual/japanese_ner_pos_and_tokenization.ipynb)
The WordSegmenter segments languages without any rule-based tokenization such as Chinese, Japanese, or Korean
Expand Down Expand Up @@ -1098,7 +1098,7 @@ df
| ませ|
| ん|


</div><div class="h3-box" markdown="1">

## Translation
[Translation example](https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/component_examples/sequence2sequence/translation_demo.ipynb)
Expand Down Expand Up @@ -1129,17 +1129,13 @@ df
|-----------|--------------|
|Billy likes to go to the mall every sunday | Billy geht gerne jeden Sonntag ins Einkaufszentrum|





</div><div class="h3-box" markdown="1">

## T5
[Example of every T5 task](https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/component_examples/sequence2sequence/T5_tasks_summarize_question_answering_and_more.ipynb)
### Overview of every task available with T5
[The T5 model](https://arxiv.org/pdf/1910.10683.pdf) is trained on various datasets for 17 different tasks which fall into 8 categories.


1. Text summarization
2. Question answering
3. Translation
Expand All @@ -1149,6 +1145,8 @@ df
7. Sentence Completion
8. Word sense disambiguation

</div><div class="h3-box" markdown="1">

### Every T5 Task with explanation:


Expand Down Expand Up @@ -1177,8 +1175,7 @@ df
- [Every T5 Task example notebook](https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/component_examples/sequence2sequence/T5_tasks_summarize_question_answering_and_more.ipynb) to see how to use every T5 Task.
- [T5 Open and Closed Book question answering notebook](https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/component_examples/sequence2sequence/T5_question_answering.ipynb)



</div><div class="h3-box" markdown="1">

## Text Summarization
[Summarization example](https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/component_examples/sequence2sequence/T5_tasks_summarize_question_answering_and_more.ipynb)
Expand Down Expand Up @@ -1208,6 +1205,7 @@ pipe.predict(data)
|------------------|-------|
| manchester united face newcastle in the premier league on wednesday . louis van gaal's side currently sit two points clear of liverpool in fourth . the belgian duo took to the dance floor on monday night with some friends . | the belgian duo took to the dance floor on monday night with some friends . manchester united face newcastle in the premier league on wednesday . red devils will be looking for just their second league away win in seven . louis van gaal’s side currently sit two points clear of liverpool in fourth . |

</div><div class="h3-box" markdown="1">

## Binary Sentence similarity/ Paraphrasing
[Binary sentence similarity example](https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/component_examples/sequence2sequence/T5_tasks_summarize_question_answering_and_more.ipynb)
Expand Down Expand Up @@ -1242,6 +1240,7 @@ t5.predict(data)
|We acted because we saw the existing evidence in a new light , through the prism of our experience on 11 September , " Rumsfeld said .| Rather , the US acted because the administration saw " existing evidence in a new light , through the prism of our experience on September 11 " . | equivalent |
| I like to eat peanutbutter for breakfast| I like to play football | not_equivalent |

</div><div class="h3-box" markdown="1">

### How to configure T5 task for MRPC and pre-process text
`.setTask('mrpc sentence1:)` and prefix second sentence with `sentence2:`
Expand All @@ -1254,7 +1253,7 @@ sentence1: We acted because we saw the existing evidence in a new light , throug
sentence2: Rather , the US acted because the administration saw " existing evidence in a new light , through the prism of our experience on September 11",
```


</div><div class="h3-box" markdown="1">

## Regressive Sentence similarity/ Paraphrasing

Expand Down Expand Up @@ -1299,12 +1298,12 @@ t5.predict(data)
|What was it like in Ancient rome? | What was Ancient rome like?| 5.0 |
|What was live like as a King in Ancient Rome?? | What is it like to live in Rome? | 3.2 |

</div><div class="h3-box" markdown="1">

### How to configure T5 task for stsb and pre-process text
`.setTask('stsb sentence1:)` and prefix second sentence with `sentence2:`



</div><div class="h3-box" markdown="1">

### Example pre-processed input for T5 STSB - Regressive semantic sentence similarity

Expand All @@ -1314,9 +1313,7 @@ sentence1: What attributes would have made you highly desirable in ancient Rome?
sentence2: How I GET OPPERTINUTY TO JOIN IT COMPANY AS A FRESHER?',
```




</div><div class="h3-box" markdown="1">

## Grammar Checking
[Grammar checking with T5 example](https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/component_examples/sequence2sequence/T5_tasks_summarize_question_answering_and_more.ipynb))
Expand All @@ -1339,10 +1336,7 @@ pipe.predict(data)
| Anna and Mike is going skiing and they is liked is | unacceptable |
| Anna and Mike like to dance | acceptable |





</div><div class="h3-box" markdown="1">

## Open book question answering
[T5 Open and Closed Book question answering tutorial](https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/component_examples/sequence2sequence/T5_question_answering.ipynb)
Expand Down Expand Up @@ -1420,7 +1414,7 @@ surged 5% | How did Alibaba stocks react? |
100 rural teachers | Whom did Jack Ma meet? |
Chinese regulators |Who did Jack Ma hide from?|


</div><div class="h3-box" markdown="1">

## Closed book question answering
[T5 Open and Closed Book question answering tutorial](https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/component_examples/sequence2sequence/T5_question_answering.ipynb)
Expand Down Expand Up @@ -1454,5 +1448,4 @@ nlu.load('en.t5').predict('What is the capital of Germany?')
>>> Berlin
```



</div></div>
45 changes: 18 additions & 27 deletions docs/en/examples_healthcare.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,7 @@ key: docs-examples-hc
permalink: /docs/en/examples_hc
modify_date: "2019-05-16"
---
<div class="main-docs" markdown="1">

<div class="h3-box" markdown="1">
<div class="main-docs" markdown="1"><div class="h3-box" markdown="1">

## Usage examples of NLU.load()
The following examples demonstrate how to use nlu's load api accompanied by the outputs generated by it.
Expand All @@ -18,9 +16,7 @@ You need to pass one NLU reference to the load method.
You can also pass multiple whitespace separated references.
[You can find all NLU references here](https://nlu.johnsnowlabs.com/docs/en/spellbook)




</div><div class="h3-box" markdown="1">

## Medical Named Entity Recognition (NER)
[Medical NER tutorial notebook](https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/healthcare/medical_named_entity_recognition/overview_medical_entity_recognizers.ipynb)
Expand Down Expand Up @@ -48,6 +44,8 @@ df = nlu.load('med_ner.jsl.wip.clinical en.resolve_chunk.cpt_clinical').predict(

See the [Models Hub for all avaiable Entity Resolution Models](https://nlp.johnsnowlabs.com/models?task=Named+Entity+Recognition)

</div><div class="h3-box" markdown="1">

## Entity Resolution (for sentences)
[Entity Resolution tutorial notebook](https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/healthcare/entity_resolution/entity_resolvers_overview.ipynb)

Expand All @@ -67,8 +65,7 @@ data = ["""He has a starvation ketosis but nothing found for significant for dry

See the [Models Hub for all avaiable Entity Resolution Models](https://nlp.johnsnowlabs.com/models?task=Entity+Resolution)



</div><div class="h3-box" markdown="1">

## Entity Resolution (for chunks)
[Entity Resolution tutorial notebook](https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/healthcare/entity_resolution/entity_resolvers_overview.ipynb)
Expand All @@ -95,10 +92,9 @@ df = nlu.load('med_ner.jsl.wip.clinical en.resolve_chunk.cpt_clinical').predict(
| for 2 days | Duration | 0.5479 | 35390 | for 2 days | 2.3929 | 1 | 35390 | 2.3929 | 0.22 |




See the [Models Hub for all avaiable Entity Resolution Models](https://nlp.johnsnowlabs.com/models?task=Entity+Resolution)

</div><div class="h3-box" markdown="1">

## Relation Extraction
[Relation Extraction tutorial notebook](https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/healthcare/relation_extraction/overview_relation.ipynb)
Expand Down Expand Up @@ -150,12 +146,7 @@ df = nlu.load('en.med_ner.jsl.wip.clinical.greedy en.relation').predict(data)
| MRI demonstrated infarction in the upper brain stem , left cerebellum and right basil ganglia" | 0 | Internal_organ_or_component | Internal_organ_or_component | cerebellum | basil ganglia | 0.975779 | ['MRI', 'infarction', 'upper', 'brain stem', 'left', 'cerebellum', 'right', 'basil ganglia'] | ['Test', 'Disease_Syndrome_Disorder', 'Direction', 'Internal_organ_or_component', 'Direction', 'Internal_organ_or_component', 'Direction', 'Internal_organ_or_component'] | ['0.9979', '0.5062', '0.2152', '0.2636', '0.4775', '0.8135', '0.5086', '0.3236'] |
| MRI demonstrated infarction in the upper brain stem , left cerebellum and right basil ganglia" | 1 | Direction | Internal_organ_or_component | right | basil ganglia | 0.999613 | ['MRI', 'infarction', 'upper', 'brain stem', 'left', 'cerebellum', 'right', 'basil ganglia'] | ['Test', 'Disease_Syndrome_Disorder', 'Direction', 'Internal_organ_or_component', 'Direction', 'Internal_organ_or_component', 'Direction', 'Internal_organ_or_component'] | ['0.9979', '0.5062', '0.2152', '0.2636', '0.4775', '0.8135', '0.5086', '0.3236'] |







</div><div class="h3-box" markdown="1">

## Assertion
[Assertion tutorial notebook](https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/healthcare/assertion/assertion_overview.ipynb)
Expand All @@ -174,6 +165,7 @@ assert_df = nlu.load('en.med_ner.clinical en.assert ').predict(data)

See the [Models Hub for all avaiable Assertion Models](https://nlp.johnsnowlabs.com/models?task=Assertion+Status)

</div><div class="h3-box" markdown="1">

## De-Identification
[De-Identification tutorial notebook](https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/healthcare/de_identification/DeIdentification_model_overview.ipynb)
Expand All @@ -192,6 +184,7 @@ df = nlu.load('de_identify').predict(data)

See the [Models Hub for all avaiable De-Identification Models](https://nlp.johnsnowlabs.com/models?task=De-identification)

</div><div class="h3-box" markdown="1">

## Drug Normalizer
[Drug Normalizer tutorial notebook](https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/healthcare/drug_normalization/drug_norm.ipynb)
Expand All @@ -217,6 +210,7 @@ nlu.load('norm_drugs').predict(data)
| interferon alfa - 2b 10000000 unt ( 1 ml ) injection | interferon alfa-2b 10 million unit ( 1 ml ) injec |
| Sodium Chloride / Potassium Chloride 13 bag | Sodium Chloride/Potassium Chloride 13bag |

</div><div class="h3-box" markdown="1">

## Rule based NER with Context Matcher
[Rule based NER with context matching tutorial notebook](https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/Training/rule_based_named_entity_recognition_and_resolution/rule_based_NER_and_resolution_with_context_matching.ipynb)
Expand Down Expand Up @@ -270,6 +264,8 @@ gender_NER_pipe.predict(sample_text)
| girl | 0.13 |
| girl | 0.13 |

</div><div class="h3-box" markdown="1">

### Context Matcher Parameters
You can define the following parameters in your rules.json file to define the entities to be matched

Expand All @@ -290,27 +286,28 @@ You can define the following parameters in your rules.json file to define the en
| completeMatchRegex | `Optional[str]`| Wether to use complete or partial matching, either `"true"` or `"false"` |
| ruleScope | `str` | currently only `sentence` supported |





</div><div class="h3-box" markdown="1">

## Authorize access to licensed features and install healthcare dependencies
You need a set of **credentials** to access the licensed healthcare features.
[You can grab one here](https://www.johnsnowlabs.com/spark-nlp-try-free/)

</div><div class="h3-box" markdown="1">

### Automatically Authorize Google Colab via JSON file
By default, nlu checks `/content/spark_nlp_for_healthcare.json` on google colabe enviroments for a `spark_nlp_for_healthcare.json` file that you recieve via E-mail from us.
If you upload the `spark_nlp_for_healthcare.json` file to the standard colab directory, `nlu.load()` will automatically find it and authorize your enviroment.

</div><div class="h3-box" markdown="1">

### Authorize anywhere via providing via JSON file
You can specify the location of your `spark_nlp_for_healthcare.json` like this :
```python
path = '/path/to/spark_nlp_for_healthcare.json'
nlu.auth(path).load('licensed_model').predict(data)
```

</div><div class="h3-box" markdown="1">

### Authorize via providing String parameters
```python
Expand All @@ -321,12 +318,6 @@ AWS_SECRET_ACCESS_KEY = 'YOUR_SECRETS'
JSL_SECRET = 'YOUR_SECRETS'

nlu.auth(SPARK_NLP_LICENSE,AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,JSL_SECRET)


```






</div>
Loading