Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add vertexai embeddings #2693

Merged
merged 38 commits into from
Mar 28, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
381ccac
build(deps): bump version for security patches
MthwRobinson Mar 15, 2024
18d3817
Merge branch 'main' into deps/security-bump
MthwRobinson Mar 15, 2024
27c1cab
pin unstructured client
MthwRobinson Mar 15, 2024
04e2058
Merge branch 'deps/security-bump' of github.com:Unstructured-IO/unstr…
MthwRobinson Mar 15, 2024
a2a1649
add vertexai with testing on pinecone
ahmetmeleq Mar 26, 2024
ce53a12
Merge branch 'main' into ahmet/vertex-embed
ahmetmeleq Mar 26, 2024
d472223
make tidy
ahmetmeleq Mar 26, 2024
e706f54
Merge branch 'ahmet/vertex-embed' of https://github.com/Unstructured-…
ahmetmeleq Mar 26, 2024
0ac052c
fix import on unit test
ahmetmeleq Mar 26, 2024
f5a6cdc
Merge branch 'deps/security-bump' into ahmet/vertex-embed
ahmetmeleq Mar 26, 2024
e09156f
requirements update, pip compile for embed modules
ahmetmeleq Mar 26, 2024
43bcdfc
add vertexai integration test
ahmetmeleq Mar 26, 2024
66f1a2c
add octoai test
ahmetmeleq Mar 26, 2024
dbaaf6f
dependency updates for embedding modules
ahmetmeleq Mar 26, 2024
7b2368b
shellcheck
ahmetmeleq Mar 26, 2024
3f7c20c
shfmt
ahmetmeleq Mar 26, 2024
9208ea8
Revert "Merge branch 'deps/security-bump' into ahmet/vertex-embed"
ahmetmeleq Mar 26, 2024
5ec18de
fix typo, fix extra name
ahmetmeleq Mar 26, 2024
827eba3
update test-ingest-src
ahmetmeleq Mar 27, 2024
658de79
change extra name for octoai
ahmetmeleq Mar 27, 2024
95a619e
parametrized api-key
ahmetmeleq Mar 27, 2024
86cc766
Merge branch 'main' into ahmet/vertex-embed
ahmetmeleq Mar 27, 2024
21c08d8
version
ahmetmeleq Mar 27, 2024
c81d3b8
update docs based on parametrized api_key
ahmetmeleq Mar 27, 2024
dffde65
testing to invalidate github cache
ahmetmeleq Mar 27, 2024
975b8f0
debugging cache
ahmetmeleq Mar 27, 2024
cd1cb7a
try healing the cache via a save without load
ahmetmeleq Mar 28, 2024
6a007cc
re-enable loads
ahmetmeleq Mar 28, 2024
3805622
add api_key to mock test
ahmetmeleq Mar 28, 2024
78c69a1
add credentials cleanup
ahmetmeleq Mar 28, 2024
2866f42
save creds to tmp rather than manual cleanup
ahmetmeleq Mar 28, 2024
dd47e56
heal cache
ahmetmeleq Mar 28, 2024
3e489ff
Revert "heal cache"
ahmetmeleq Mar 28, 2024
5b278bf
working example in examples with comments updated
ahmetmeleq Mar 28, 2024
7587fd3
working example in docs with comments updated
ahmetmeleq Mar 28, 2024
4392700
vectara fix
ahmetmeleq Mar 28, 2024
0cc52d7
Merge branch 'main' into ahmet/vertex-embed
ahmetmeleq Mar 28, 2024
b639fc1
version
ahmetmeleq Mar 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
try healing the cache via a save without load
  • Loading branch information
ahmetmeleq committed Mar 28, 2024
commit cd1cb7afcd220460b975c484e34d58bcb0237abb
11 changes: 10 additions & 1 deletion .github/actions/base-cache/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,13 @@ runs:
python${{ inputs.python-version }} -m venv .venv
source .venv/bin/activate
[ ! -d "$NLTK_DATA" ] && mkdir "$NLTK_DATA"
make install-ci
make install-ci
- name: Save Cache
if: steps.virtualenv-cache-restore.outputs.cache-hit != 'true'
id: virtualenv-cache-save
uses: actions/cache/save@v3
with:
path: |
.venv
nltk_data
key: unstructured-${{ runner.os }}-${{ inputs.python-version }}-${{ hashFiles('requirements/*.txt') }}
27 changes: 23 additions & 4 deletions .github/actions/base-ingest-cache/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,20 +11,39 @@ inputs:
runs:
using: "composite"
steps:
- name: Check for/restore ingest cache
uses: actions/cache/restore@v3
id: ingest-virtualenv-cache-restore
with:
path: |
.venv
nltk_data
key: unstructured-ingest-${{ runner.os }}-${{ inputs.python-version }}-${{ hashFiles('requirements/ingest/*.txt') }}
lookup-only: ${{ inputs.check-only }}
- name: Restore base virtual environment
if: 'true'
if: steps.ingest-virtualenv-cache-restore.outputs.cache-hit != 'true'
uses: ./.github/actions/base-cache
with:
python-version: ${{ inputs.python-version }}
- name: Set up Python ${{ inputs.python-version }}
if: 'true'
if: steps.ingest-virtualenv-cache-restore.outputs.cache-hit != 'true'
uses: actions/setup-python@v4
with:
python-version: ${{ inputs.python-version }}
- name: Setup virtual environment (no cache hit)
if: 'true'
if: steps.ingest-virtualenv-cache-restore.outputs.cache-hit != 'true'
shell: bash
run: |
python${{ inputs.python-version }} -m venv .venv
source .venv/bin/activate
make install-all-ingest
make install-all-ingest
- name: Save Ingest Cache
if: steps.ingest-virtualenv-cache-restore.outputs.cache-hit != 'true'
id: ingest-virtualenv-cache-save
uses: actions/cache/save@v3
with:
path: |
.venv
nltk_data
key: unstructured-ingest-${{ runner.os }}-${{ inputs.python-version }}-${{ hashFiles('requirements/ingest/*.txt') }}

Loading