Various server/program improvements (frictionlessdata#1459)

* Bootstrapped sql mode * Support packages * Renamed to sql * Added more source guessing in Resource * Improved tests * Added LocalAdapter * Fixed problems * Fixed catalog * Reworked extract * Renamed sql to query * Reworked program.extract * Recover csv support for program.extract * Fixed extract * Limit extract by default * Fixed some tests * Recovered extract tests * Fixed read_package signature * Fixed github read_package * Improved extract * Fixed ckan read_package * Skip non-tabular for package extart * Finalized extract * Fixed some tests * Fixed ckan formats * Fixed linting * Improved resoruce detection * Bootstrapped CLI docs * Fixed ckan docs headings * Make resource.tabular static * Bootstrapped propert TableResoure * Added non-tabular validate * Updated package methods * Added PackageResource validate * Added MetadataResource descriptr * Implemeted resource classes * Rebased actions.validate on Resource * Hide some console commands * Rebased program.validate on Resource * Improved describe output * Improved cli * Improved cli error messages * Rebased validate on rich * Don't validate metadata by default * Improved MetadataResource * Rebased describe on Resource * Updated describe methods * Support describe's name arg * Rebased program.describe on Resource * Rebased actions.transform on Resource * Rebased program.transform on Resource * Added list/index placeholders * Implemented list action * Implemented program.list * Rebased describe on rich * Improved validate in CLI * Improved extract * Rebased program.describe on actions * Rebased program.list on actions * Support Schema from descriptor shortcut * Allow from_descriptor shortcut for Dialect * Implemented actions.index * Bootstrapped program.script * Support indexing packages in CLI * Reworked program.query * Improved index/query * Implemented program.script * Move read_catalog to experimental section * Implemented program.explore * Support name for list/etc * Lowercase df name * Improve indexing progress * Removed script from blog * Fixed linting * Added "$frictionless" prop to data package * Added program.publish * Added failing test for 1444 * Fixed 1444 group char * Updated standards * Implemented convert command * Fixed linting * Fixed order * Added progress/result to convert * Imporved Resource.describe * Rebased on traits * Fixed type errors in tests * Fixed extract tests * Updated csv/excel tests * Updated gsheets/html/inling/json/ods/pandas tests * Updated schemes tests * Rebased some tests on TableResource * Fixed linting * Fixed package tests * Fixed package tests * Fixed portals tests * Fixed project * Fixed steps tests * Fixed linting * Fixed program.describe tests * Fixed program.extract tests * Fixed program.validate tests * Moved tabular attributes to TableResource * Fixed tests * Improved methods order * Fixed some type errors * Fixed type errors for portal tests * Fixed formats/steps tests type errors * Fixed steps type errors * Fixed tests packages type errors * Fixed type errors in tests * Fixed type errors * Fixed tests * Fixed tests * Moved Analyzer to its own module * Moved Indexer to its own module * Moved transformer to its own module * Moved validator to its own module * Fixed name/datatype optional problem * Fixed actions not backward-compatible * Fixed docs * Fixed linting * Added docs for new CLI commands * Updated CLI commands' docs * Added resources docs
fjuniorr · Mar 15, 2023 · eb7d12c · eb7d12c
1 parent 6ef2296
commit eb7d12c
Show file tree

Hide file tree

Showing 337 changed files with 6,788 additions and 4,449 deletions.
diff --git a/assets/explore.png b/assets/explore.png
diff --git a/assets/publish.png b/assets/publish.png
diff --git a/assets/query.png b/assets/query.png
diff --git a/assets/script.png b/assets/script.png
diff --git a/blog/2022/08-22-frictionless-framework-v5.md b/blog/2022/08-22-frictionless-framework-v5.md
@@ -29,7 +29,7 @@ csv:
   delimiter: ';'
 ```
 
-```python script tabs=Python
+```python tabs=Python
 from frictionless import Dialect, Control, formats
 
 dialect = Dialect(header=True, header_rows=[2, 3], comment_char='#')
@@ -44,7 +44,7 @@ header: true
 delimiter: ';'
 ```
 
-```python script tabs=Python
+```python tabs=Python
 from frictionless import Dialect, Control, formats
 
 dialect = Dialect.from_descriptor({"header": True, "delimiter": ';'})
@@ -68,7 +68,7 @@ skipErrors:
   - duplicate-label
 ```
 
-```python script tabs=Python
+```python tabs=Python
 from frictionless import Checklist, checks
 
 checklist = Checklist(
@@ -106,7 +106,7 @@ steps:
     value: v5
 ```
 
-```python script tabs=Python
+```python tabs=Python
 from frictionless import Pipeline, steps
 
 pipeline = Pipeline(
@@ -175,7 +175,7 @@ frictionless transform resource.yaml  # will use the pipeline above
 
 There are minor changes in the `stats` property. Now it uses named keys to simplify hash distinction (md5/sha256 are calculated by default and it's not possible to change for performance reasons as it was in v4):
 
-```python script tabs=Python
+```python tabs=Python
 from frictionless import describe
 
 resource = describe('table.csv', stats=True)
@@ -224,7 +224,7 @@ Read an article about [Catalog Class](../../docs/framework/catalog.html) for mor
 
 Detector is now a metadata class (it wasn't in v4) so it can be saved and shared as other metadata classes:
 
-```python script tabs=Python
+```python tabs=Python
 from frictionless import Detector
 
 detector = Detector(sample_size=1000)
@@ -249,7 +249,7 @@ Read an article about [Inquiry Class](../../docs/framework/inquiry.html) for mor
 
 The Report concept has been significantly simplified by removing the `resource` property from `reportTask`. It's been replaced by `name/type/place/labels` properties. Also `report.time` is now `report.stats.seconds`. The `report/reportTask.warnings: List[str]` have been added to provide non-error information like reached limits:
 
-```bash script tabs=CLI output=yaml
+```bash tabs=CLI output=yaml
 frictionless validate table.csv --yaml
 ```
 

diff --git a/data/capital.package.yaml b/data/capital.package.yaml
@@ -1,3 +1,4 @@
+$frictionless: package/v2
 resources:
   - name: capital-invalid
     type: table

diff --git a/data/cassettes/test_extract_description_option_issue_1362.yaml b/data/cassettes/test_extract_description_option_issue_1362.yaml
diff --git a/data/cassettes/test_resource_compression_remote_csv_zip.yaml b/data/cassettes/test_resource_compression_remote_csv_zip.yaml
@@ -36,17 +36,17 @@ interactions:
       Content-Type:
       - application/zip
       Date:
-      - Tue, 09 Feb 2021 08:44:31 GMT
+      - Thu, 02 Mar 2023 10:51:55 GMT
       ETag:
       - W/"8d94a272dee43d9209f7ec9e49b9e266c6e6f474c2c796c7faa4b068ecad96be"
       Expires:
-      - Tue, 09 Feb 2021 08:49:31 GMT
+      - Thu, 02 Mar 2023 10:56:55 GMT
       Source-Age:
       - '0'
       Strict-Transport-Security:
       - max-age=31536000
       Vary:
-      - Authorization,Accept-Encoding
+      - Authorization,Accept-Encoding,Origin
       Via:
       - 1.1 varnish
       X-Cache:
@@ -56,15 +56,157 @@ interactions:
       X-Content-Type-Options:
       - nosniff
       X-Fastly-Request-ID:
-      - 06eaae1c7cc04b4490cf4b7a9c914cb0b8616387
+      - 8d10030f3212ca6f3028e91a3a9972ef8a656893
       X-Frame-Options:
       - deny
       X-GitHub-Request-Id:
-      - 7C9C:E8DD:D1540:E0910:60224B6B
+      - 197A:C879:555668:60205B:64007FCB
       X-Served-By:
-      - cache-fra19142-FRA
+      - cache-lis1490044-LIS
       X-Timer:
-      - S1612860271.842178,VS0,VE168
+      - S1677754316.707239,VS0,VE256
+      X-XSS-Protection:
+      - 1; mode=block
+    status:
+      code: 200
+      message: OK
+- request:
+    body: null
+    headers:
+      Accept:
+      - '*/*'
+      Accept-Encoding:
+      - gzip, deflate
+      Connection:
+      - keep-alive
+      User-Agent:
+      - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML,
+        like Gecko) Chrome/54.0.2840.87 Safari/537.36
+    method: GET
+    uri: https://raw.githubusercontent.com/frictionlessdata/tabulator-py/master/data/table.csv.zip
+  response:
+    body:
+      string: !!binary |
+        UEsDBAoAAAAAAPJQZUkBbkCNHgAAAB4AAAAJABwAdGFibGUuY3N2VVQJAAM3hR1Y0J/oWXV4CwAB
+        BOgDAAAE6AMAAGlkLG5hbWUKMSxlbmdsaXNoCjIs5Lit5Zu95Lq6ClBLAQIeAwoAAAAAAPJQZUkB
+        bkCNHgAAAB4AAAAJABgAAAAAAAEAAAC0gQAAAAB0YWJsZS5jc3ZVVAUAAzeFHVh1eAsAAQToAwAA
+        BOgDAABQSwUGAAAAAAEAAQBPAAAAYQAAAAAA
+    headers:
+      Accept-Ranges:
+      - bytes
+      Access-Control-Allow-Origin:
+      - '*'
+      Cache-Control:
+      - max-age=300
+      Connection:
+      - keep-alive
+      Content-Length:
+      - '198'
+      Content-Security-Policy:
+      - default-src 'none'; style-src 'unsafe-inline'; sandbox
+      Content-Type:
+      - application/zip
+      Date:
+      - Thu, 02 Mar 2023 10:51:56 GMT
+      ETag:
+      - W/"8d94a272dee43d9209f7ec9e49b9e266c6e6f474c2c796c7faa4b068ecad96be"
+      Expires:
+      - Thu, 02 Mar 2023 10:56:56 GMT
+      Source-Age:
+      - '0'
+      Strict-Transport-Security:
+      - max-age=31536000
+      Vary:
+      - Authorization,Accept-Encoding,Origin
+      Via:
+      - 1.1 varnish
+      X-Cache:
+      - HIT
+      X-Cache-Hits:
+      - '1'
+      X-Content-Type-Options:
+      - nosniff
+      X-Fastly-Request-ID:
+      - a73fc44c18733aa908f518f9e7e9a712574dfa50
+      X-Frame-Options:
+      - deny
+      X-GitHub-Request-Id:
+      - 197A:C879:555668:60205B:64007FCB
+      X-Served-By:
+      - cache-lis1490043-LIS
+      X-Timer:
+      - S1677754316.268387,VS0,VE1
+      X-XSS-Protection:
+      - 1; mode=block
+    status:
+      code: 200
+      message: OK
+- request:
+    body: null
+    headers:
+      Accept:
+      - '*/*'
+      Accept-Encoding:
+      - gzip, deflate
+      Connection:
+      - keep-alive
+      User-Agent:
+      - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML,
+        like Gecko) Chrome/54.0.2840.87 Safari/537.36
+    method: GET
+    uri: https://raw.githubusercontent.com/frictionlessdata/tabulator-py/master/data/table.csv.zip
+  response:
+    body:
+      string: !!binary |
+        UEsDBAoAAAAAAPJQZUkBbkCNHgAAAB4AAAAJABwAdGFibGUuY3N2VVQJAAM3hR1Y0J/oWXV4CwAB
+        BOgDAAAE6AMAAGlkLG5hbWUKMSxlbmdsaXNoCjIs5Lit5Zu95Lq6ClBLAQIeAwoAAAAAAPJQZUkB
+        bkCNHgAAAB4AAAAJABgAAAAAAAEAAAC0gQAAAAB0YWJsZS5jc3ZVVAUAAzeFHVh1eAsAAQToAwAA
+        BOgDAABQSwUGAAAAAAEAAQBPAAAAYQAAAAAA
+    headers:
+      Accept-Ranges:
+      - bytes
+      Access-Control-Allow-Origin:
+      - '*'
+      Cache-Control:
+      - max-age=300
+      Connection:
+      - keep-alive
+      Content-Length:
+      - '198'
+      Content-Security-Policy:
+      - default-src 'none'; style-src 'unsafe-inline'; sandbox
+      Content-Type:
+      - application/zip
+      Date:
+      - Thu, 02 Mar 2023 10:51:56 GMT
+      ETag:
+      - W/"8d94a272dee43d9209f7ec9e49b9e266c6e6f474c2c796c7faa4b068ecad96be"
+      Expires:
+      - Thu, 02 Mar 2023 10:56:56 GMT
+      Source-Age:
+      - '0'
+      Strict-Transport-Security:
+      - max-age=31536000
+      Vary:
+      - Authorization,Accept-Encoding,Origin
+      Via:
+      - 1.1 varnish
+      X-Cache:
+      - HIT
+      X-Cache-Hits:
+      - '2'
+      X-Content-Type-Options:
+      - nosniff
+      X-Fastly-Request-ID:
+      - ee9bc8af05bb8687815be293c635e4dabc8fc0ac
+      X-Frame-Options:
+      - deny
+      X-GitHub-Request-Id:
+      - 197A:C879:555668:60205B:64007FCB
+      X-Served-By:
+      - cache-lis1490043-LIS
+      X-Timer:
+      - S1677754316.336172,VS0,VE0
       X-XSS-Protection:
       - 1; mode=block
     status: