Jul 08 22:20:44 szparag machine-learning[119064]: [2025-07-08 22:20:44 +0200] [119064] [ERROR] Exception in ASGI application Jul 08 22:20:44 szparag machine-learning[119064]: Traceback (most recent call last): Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/9iyhk28imp1qz2q169abd4chw6p9l1rs-python3.13-uvicorn-0.34.2/lib/python3.13/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi Jul 08 22:20:44 szparag machine-learning[119064]: result = await app( # type: ignore[func-returns-value] Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: self.scope, self.receive, self.send Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: ) Jul 08 22:20:44 szparag machine-learning[119064]: ^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/9iyhk28imp1qz2q169abd4chw6p9l1rs-python3.13-uvicorn-0.34.2/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__ Jul 08 22:20:44 szparag machine-learning[119064]: return await self.app(scope, receive, send) Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/47jdsspx0vr2bbn9a8pbcgnhrsraz5y7-python3.13-fastapi-0.115.12/lib/python3.13/site-packages/fastapi/applications.py", line 1054, in __call__ Jul 08 22:20:44 szparag machine-learning[119064]: await super().__call__(scope, receive, send) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/y79haj1bl2ymq8gx67l8h206mr89p8cc-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/applications.py", line 112, in __call__ Jul 08 22:20:44 szparag machine-learning[119064]: await self.middleware_stack(scope, receive, send) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/y79haj1bl2ymq8gx67l8h206mr89p8cc-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/middleware/errors.py", line 187, in __call__ Jul 08 22:20:44 szparag machine-learning[119064]: raise exc Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/y79haj1bl2ymq8gx67l8h206mr89p8cc-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/middleware/errors.py", line 165, in __call__ Jul 08 22:20:44 szparag machine-learning[119064]: await self.app(scope, receive, _send) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/y79haj1bl2ymq8gx67l8h206mr89p8cc-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ Jul 08 22:20:44 szparag machine-learning[119064]: await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/y79haj1bl2ymq8gx67l8h206mr89p8cc-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app Jul 08 22:20:44 szparag machine-learning[119064]: raise exc Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/y79haj1bl2ymq8gx67l8h206mr89p8cc-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app Jul 08 22:20:44 szparag machine-learning[119064]: await app(scope, receive, sender) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/y79haj1bl2ymq8gx67l8h206mr89p8cc-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/routing.py", line 714, in __call__ Jul 08 22:20:44 szparag machine-learning[119064]: await self.middleware_stack(scope, receive, send) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/y79haj1bl2ymq8gx67l8h206mr89p8cc-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/routing.py", line 734, in app Jul 08 22:20:44 szparag machine-learning[119064]: await route.handle(scope, receive, send) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/y79haj1bl2ymq8gx67l8h206mr89p8cc-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/routing.py", line 288, in handle Jul 08 22:20:44 szparag machine-learning[119064]: await self.app(scope, receive, send) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/y79haj1bl2ymq8gx67l8h206mr89p8cc-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/routing.py", line 76, in app Jul 08 22:20:44 szparag machine-learning[119064]: await wrap_app_handling_exceptions(app, request)(scope, receive, send) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/y79haj1bl2ymq8gx67l8h206mr89p8cc-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app Jul 08 22:20:44 szparag machine-learning[119064]: raise exc Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/y79haj1bl2ymq8gx67l8h206mr89p8cc-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app Jul 08 22:20:44 szparag machine-learning[119064]: await app(scope, receive, sender) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/y79haj1bl2ymq8gx67l8h206mr89p8cc-python3.13-starlette-0.46.2/lib/python3.13/site-packages/starlette/routing.py", line 73, in app Jul 08 22:20:44 szparag machine-learning[119064]: response = await f(request) Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/47jdsspx0vr2bbn9a8pbcgnhrsraz5y7-python3.13-fastapi-0.115.12/lib/python3.13/site-packages/fastapi/routing.py", line 301, in app Jul 08 22:20:44 szparag machine-learning[119064]: raw_response = await run_endpoint_function( Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: ...<3 lines>... Jul 08 22:20:44 szparag machine-learning[119064]: ) Jul 08 22:20:44 szparag machine-learning[119064]: ^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/47jdsspx0vr2bbn9a8pbcgnhrsraz5y7-python3.13-fastapi-0.115.12/lib/python3.13/site-packages/fastapi/routing.py", line 212, in run_endpoint_function Jul 08 22:20:44 szparag machine-learning[119064]: return await dependant.call(**values) Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/s6nqlv329syk6kkhkh4fycyhqfi62xlp-immich-machine-learning-1.135.3/lib/python3.13/site-packages/immich_ml/main.py", line 177, in predict Jul 08 22:20:44 szparag machine-learning[119064]: response = await run_inference(inputs, entries) Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/s6nqlv329syk6kkhkh4fycyhqfi62xlp-immich-machine-learning-1.135.3/lib/python3.13/site-packages/immich_ml/main.py", line 200, in run_inference Jul 08 22:20:44 szparag machine-learning[119064]: await asyncio.gather(*[_run_inference(entry) for entry in without_deps]) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/s6nqlv329syk6kkhkh4fycyhqfi62xlp-immich-machine-learning-1.135.3/lib/python3.13/site-packages/immich_ml/main.py", line 194, in _run_inference Jul 08 22:20:44 szparag machine-learning[119064]: model = await load(model) Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/s6nqlv329syk6kkhkh4fycyhqfi62xlp-immich-machine-learning-1.135.3/lib/python3.13/site-packages/immich_ml/main.py", line 238, in load Jul 08 22:20:44 szparag machine-learning[119064]: return await run(_load, model) Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/s6nqlv329syk6kkhkh4fycyhqfi62xlp-immich-machine-learning-1.135.3/lib/python3.13/site-packages/immich_ml/main.py", line 213, in run Jul 08 22:20:44 szparag machine-learning[119064]: return await asyncio.get_running_loop().run_in_executor(thread_pool, partial_func) Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/sd81bvmch7njdpwx3lkjslixcbj5mivz-python3-3.13.4/lib/python3.13/concurrent/futures/thread.py", line 59, in run Jul 08 22:20:44 szparag machine-learning[119064]: result = self.fn(*self.args, **self.kwargs) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/s6nqlv329syk6kkhkh4fycyhqfi62xlp-immich-machine-learning-1.135.3/lib/python3.13/site-packages/immich_ml/main.py", line 225, in _load Jul 08 22:20:44 szparag machine-learning[119064]: model.load() Jul 08 22:20:44 szparag machine-learning[119064]: ~~~~~~~~~~^^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/s6nqlv329syk6kkhkh4fycyhqfi62xlp-immich-machine-learning-1.135.3/lib/python3.13/site-packages/immich_ml/models/base.py", line 51, in load Jul 08 22:20:44 szparag machine-learning[119064]: self.download() Jul 08 22:20:44 szparag machine-learning[119064]: ~~~~~~~~~~~~~^^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/s6nqlv329syk6kkhkh4fycyhqfi62xlp-immich-machine-learning-1.135.3/lib/python3.13/site-packages/immich_ml/models/base.py", line 44, in download Jul 08 22:20:44 szparag machine-learning[119064]: self._download() Jul 08 22:20:44 szparag machine-learning[119064]: ~~~~~~~~~~~~~~^^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/s6nqlv329syk6kkhkh4fycyhqfi62xlp-immich-machine-learning-1.135.3/lib/python3.13/site-packages/immich_ml/models/base.py", line 76, in _download Jul 08 22:20:44 szparag machine-learning[119064]: snapshot_download( Jul 08 22:20:44 szparag machine-learning[119064]: ~~~~~~~~~~~~~~~~~^ Jul 08 22:20:44 szparag machine-learning[119064]: f"immich-app/{clean_name(self.model_name)}", Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: ...<2 lines>... Jul 08 22:20:44 szparag machine-learning[119064]: ignore_patterns=ignored_patterns.get(self.model_format, []), Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: ) Jul 08 22:20:44 szparag machine-learning[119064]: ^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/a6vmqz2r93d49cja8agfcf83yil6k0dr-python3.13-huggingface-hub-0.33.1/lib/python3.13/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn Jul 08 22:20:44 szparag machine-learning[119064]: return fn(*args, **kwargs) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/a6vmqz2r93d49cja8agfcf83yil6k0dr-python3.13-huggingface-hub-0.33.1/lib/python3.13/site-packages/huggingface_hub/_snapshot_download.py", line 327, in snapshot_download Jul 08 22:20:44 szparag machine-learning[119064]: thread_map( Jul 08 22:20:44 szparag machine-learning[119064]: ~~~~~~~~~~^ Jul 08 22:20:44 szparag machine-learning[119064]: _inner_hf_hub_download, Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: ...<4 lines>... Jul 08 22:20:44 szparag machine-learning[119064]: tqdm_class=tqdm_class or hf_tqdm, Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: ) Jul 08 22:20:44 szparag machine-learning[119064]: ^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/a6k02jyvhvixjnr0m0r0j93idks5f9r6-python3.13-tqdm-4.67.1/lib/python3.13/site-packages/tqdm/contrib/concurrent.py", line 69, in thread_map Jul 08 22:20:44 szparag machine-learning[119064]: return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/a6k02jyvhvixjnr0m0r0j93idks5f9r6-python3.13-tqdm-4.67.1/lib/python3.13/site-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map Jul 08 22:20:44 szparag machine-learning[119064]: return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs)) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/a6k02jyvhvixjnr0m0r0j93idks5f9r6-python3.13-tqdm-4.67.1/lib/python3.13/site-packages/tqdm/std.py", line 1181, in __iter__ Jul 08 22:20:44 szparag machine-learning[119064]: for obj in iterable: Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/sd81bvmch7njdpwx3lkjslixcbj5mivz-python3-3.13.4/lib/python3.13/concurrent/futures/_base.py", line 619, in result_iterator Jul 08 22:20:44 szparag machine-learning[119064]: yield _result_or_cancel(fs.pop()) Jul 08 22:20:44 szparag machine-learning[119064]: ~~~~~~~~~~~~~~~~~^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/sd81bvmch7njdpwx3lkjslixcbj5mivz-python3-3.13.4/lib/python3.13/concurrent/futures/_base.py", line 317, in _result_or_cancel Jul 08 22:20:44 szparag machine-learning[119064]: return fut.result(timeout) Jul 08 22:20:44 szparag machine-learning[119064]: ~~~~~~~~~~^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/sd81bvmch7njdpwx3lkjslixcbj5mivz-python3-3.13.4/lib/python3.13/concurrent/futures/_base.py", line 456, in result Jul 08 22:20:44 szparag machine-learning[119064]: return self.__get_result() Jul 08 22:20:44 szparag machine-learning[119064]: ~~~~~~~~~~~~~~~~~^^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/sd81bvmch7njdpwx3lkjslixcbj5mivz-python3-3.13.4/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result Jul 08 22:20:44 szparag machine-learning[119064]: raise self._exception Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/sd81bvmch7njdpwx3lkjslixcbj5mivz-python3-3.13.4/lib/python3.13/concurrent/futures/thread.py", line 59, in run Jul 08 22:20:44 szparag machine-learning[119064]: result = self.fn(*self.args, **self.kwargs) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/a6vmqz2r93d49cja8agfcf83yil6k0dr-python3.13-huggingface-hub-0.33.1/lib/python3.13/site-packages/huggingface_hub/_snapshot_download.py", line 301, in _inner_hf_hub_download Jul 08 22:20:44 szparag machine-learning[119064]: return hf_hub_download( Jul 08 22:20:44 szparag machine-learning[119064]: repo_id, Jul 08 22:20:44 szparag machine-learning[119064]: ...<15 lines>... Jul 08 22:20:44 szparag machine-learning[119064]: headers=headers, Jul 08 22:20:44 szparag machine-learning[119064]: ) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/a6vmqz2r93d49cja8agfcf83yil6k0dr-python3.13-huggingface-hub-0.33.1/lib/python3.13/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn Jul 08 22:20:44 szparag machine-learning[119064]: return fn(*args, **kwargs) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/a6vmqz2r93d49cja8agfcf83yil6k0dr-python3.13-huggingface-hub-0.33.1/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 988, in hf_hub_download Jul 08 22:20:44 szparag machine-learning[119064]: return _hf_hub_download_to_local_dir( Jul 08 22:20:44 szparag machine-learning[119064]: # Destination Jul 08 22:20:44 szparag machine-learning[119064]: ...<15 lines>... Jul 08 22:20:44 szparag machine-learning[119064]: local_files_only=local_files_only, Jul 08 22:20:44 szparag machine-learning[119064]: ) Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/a6vmqz2r93d49cja8agfcf83yil6k0dr-python3.13-huggingface-hub-0.33.1/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 1290, in _hf_hub_download_to_local_dir Jul 08 22:20:44 szparag machine-learning[119064]: _download_to_tmp_and_move( Jul 08 22:20:44 szparag machine-learning[119064]: ~~~~~~~~~~~~~~~~~~~~~~~~~^ Jul 08 22:20:44 szparag machine-learning[119064]: incomplete_path=paths.incomplete_path(etag), Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: ...<8 lines>... Jul 08 22:20:44 szparag machine-learning[119064]: xet_file_data=xet_file_data, Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: ) Jul 08 22:20:44 szparag machine-learning[119064]: ^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/a6vmqz2r93d49cja8agfcf83yil6k0dr-python3.13-huggingface-hub-0.33.1/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 1710, in _download_to_tmp_and_move Jul 08 22:20:44 szparag machine-learning[119064]: xet_get( Jul 08 22:20:44 szparag machine-learning[119064]: ~~~~~~~^ Jul 08 22:20:44 szparag machine-learning[119064]: incomplete_path=incomplete_path, Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: ...<3 lines>... Jul 08 22:20:44 szparag machine-learning[119064]: displayed_filename=filename, Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: ) Jul 08 22:20:44 szparag machine-learning[119064]: ^ Jul 08 22:20:44 szparag machine-learning[119064]: File "/nix/store/a6vmqz2r93d49cja8agfcf83yil6k0dr-python3.13-huggingface-hub-0.33.1/lib/python3.13/site-packages/huggingface_hub/file_download.py", line 627, in xet_get Jul 08 22:20:44 szparag machine-learning[119064]: download_files( Jul 08 22:20:44 szparag machine-learning[119064]: ~~~~~~~~~~~~~~^ Jul 08 22:20:44 szparag machine-learning[119064]: xet_download_info, Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: ...<3 lines>... Jul 08 22:20:44 szparag machine-learning[119064]: progress_updater=[progress_updater], Jul 08 22:20:44 szparag machine-learning[119064]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jul 08 22:20:44 szparag machine-learning[119064]: ) Jul 08 22:20:44 szparag machine-learning[119064]: ^ Jul 08 22:20:44 szparag machine-learning[119064]: RuntimeError: Data processing error: I/O error: Operation not permitted (os error 1)