python
Instagram Pinkoi iCHEF carousell Dcard - 在 Dcard 我們如何用 Python 打造推薦系統|陳子元|PyCon Taiwan 2023 - YouTube - Coding Is Magic - Speaker Deck - 為你自己學 PYTHON | 為你自己學 PYTHON
dev environment
# pip style
pip install -r requirements.txt
pip freeze
# 檢查相依性
python -m requests.help
# 安裝virtualenv
python -m venv {virtualenv dir}
# Test
python3 -c 'import math;print( math.sqrt( 512 * 1024 *1024 * 1024 * 0.85 / 8 ))'
# simple http server
python -m http.server
# Pretty-Printing JSON with Python's JSON Tool
curl http://api.joind.in | python3 -m json.tool
build
sudo apt install build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev wget liblzma-dev -y
curl https://pyenv.run | bash
cat > ~/.bashrc << EOF
# pyenv
export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
EOF
source ~/.bashrc
pyenv install 3.10.4
各版本
Python Developers Survey
performance
- 好影片推薦 — How Fast can Python Parse 1 Billion Rows of Data? - MyApollo
- 好文分享 — Python Big O: the time complexities of different data structures in Python - MyApollo
Layout
- Python Application Layouts: A Reference
- One-Off Script
- You just make a .py script, and it’s gravy, right? No need to install—just run the script in its directory
- Installable Single Package
- Let’s imagine that helloworld.py is still the main script to execute, but you’ve moved all helper methods to a new file called helpers.py
- Application with Internal Packages
- In larger applications, you may have one or more internal packages that are either tied together with a main runner script or that provide specific functionality to a larger library you are packaging
- One-Off Script
- Packaging a python library
Type hint
PyPI
- How to Publish an Open-Source Python Package to PyPI
- The most common formats for distribution packages are source archives and Python wheels
- setup.py + twine
python setup.py sdist bdist_wheel
twine check dist/*
- Cookiecutter
- Flit
- Poetry
- What Are Python Wheels and Why Should You Care
- Python packaging: wheels vs tarball (tar.gz)
- Packaging and distributing projects
- 打包python module 到PYPI 上
- PIP Install Local Package - PIP Install Wheel - PIP Install Packages Offline
- How can I install packages using pip according to the requirements.txt file from a local directory
- PyPI隔離措施奏效,數月阻擋逾百惡意專案 | iThome
- Installing Python packages from local file system folder to virtualenv with pip
```python=
pip install -r requirements.txt --no-index --find-links file:///tmp/packages
pyenv versions pyenv install --list pyenv install 3.10.8 pyenv global 3.10.8
### GIL - [深入 GIL: 如何寫出快速且 thread-safe 的 Python - Grok the GIL: How to write fast and thread-safe Python | louie\_lu's blog](https://blog.louie.lu/2017/05/19/%e6%b7%b1%e5%85%a5-gil-%e5%a6%82%e4%bd%95%e5%af%ab%e5%87%ba%e5%bf%ab%e9%80%9f%e4%b8%94-thread-safe-%e7%9a%84-python-grok-the-gil-how-to-write-fast-and-thread-safe-python/) ### 程式碼保護 - [如何保護你的python程式碼](https://zhuanlan.zhihu.com/p/54296517) 程式碼混淆: obfuscator - [How to obfuscate Python code effectively](https://stackoverflow.com/questions/3344115/how-to-obfuscate-python-code-effectively) - [Python Source Code Obfuscation](https://medium.com/geekculture/python-source-code-obfuscation-6b97f88a460d) - [Python package obfuscation](https://stackoverflow.com/questions/68691077/python-package-obfuscation) - [Distributing python packages protected with Cython](https://medium.com/swlh/distributing-python-packages-protected-with-cython-40fc29d84caf) - [How to Deploy a Cython Package to PyPI](https://levelup.gitconnected.com/how-to-deploy-a-cython-package-to-pypi-8217a6581f09) - [Publishing Proprietary Python Packages on PyPI Using Poetry](https://d39l7znklsxxzt.cloudfront.net/zh/blog/2021/01/19/publishing-a-proprietary-python-package-on-pypi-using-poetry/) #### virtualenv - [virtualenv的原理 | 卡瓦邦噶!](https://www.kawabangga.com/posts/3543) - [你的 Python 包都装到哪了? | Frost's Blog](https://frostming.com/2019/03-13/where-do-your-packages-go/) #### venv - [Python 虛擬環境--venv - DEV Community](https://dev.to/codemee/python-xu-ni-huan-jing-venv-nbg) - [把 Python 的 venv 移到其他機器 | Pan in the Nutshell](https://blog.pan93.com/posts/move-venv-to-other-machines/) #### pyenv - [Python 版本管理的好工具 - pyenv - MyApollo](https://myapollo.com.tw/blog/pyenv/) - [GitHub - pyenv/pyenv-virtualenv: a pyenv plugin to manage virtualenv (a.k.a. python-virtualenv)](https://github.com/pyenv/pyenv-virtualenv) - [pyenv no module named '\_sqlite3' - DEV Community](https://dev.to/waylonwalker/pyenv-no-module-named-sqlite3-3ea2?comments_sort=latest)
pyenv virtualenvs pyenv virtualenv 3.10.0 ansible-playbook-graph pyenv activate ansible-playbook-graph pyenv deactivate pyenv uninstall ansible-playbook-graph
#### poetry
- [koko's Note – Python - 取代 Pipenv 的新套件管理器 Poetry](https://note.koko.guru/posts/using-poetry-manage-python-package-environments)
- [再見了 pip!最佳 Python 套件管理器——Poetry 完全入門指南 - Code and Me](https://blog.kyomind.tw/python-poetry/)
- [Create and Publish a Python Package with Poetry](https://johnfraney.ca/blog/create-publish-python-package-poetry/)
- [Install](https://python-poetry.org/docs/#installation)
```bash=
curl -sSL https://install.python-poetry.org | python -
hatch
call by what
PyCon
Pythonic
- Pythonic 實踐:實用的 python 慣用法整理 | Mr. Opengate
- 友好的 Python:面向对象接口 | Frost's Blog
- 友好的 Python:扩展友好 | Frost's Blog
error
- python - Python3: ImportError: No module named '_ctypes' when using Value from module multiprocessing - Stack Overflow
- The Ultimate Guide to Error Handling in Python - miguelgrinberg.com
pacakge
Python defines two types of packages, regular packages and namespace packages. Regular packages are traditional packages as they existed in Python 3.2 and earlier. A regular package is typically implemented as a directory containing an __init__.py
offiline installation
- https://superuser.com/questions/1523218/how-to-install-python-packages-with-all-dependencies-offline-via-pip3
參考
- python docs
- __init__.py
- Python Modules and Packages – An Introduction
- This can be used for execution of package initialization code, such as initialization of package-level data
- __init__.py can also be used to effect automatic importing of modules from a package.
- What is __init__.py for?
- What's your opinion on what to include in __init__.py ?
- Python Modules and Packages – An Introduction
- organizie
- Can someone explain __all__ in Python?
- 撥開 Python, pip, site-packages 的藍色蜘蛛網
- :star: How to create a Python package in 2022 | Mathspp
module
module vs class
- Why use python classes over modules with functions
- Difference between Module and Class in Python
- choosing between Modules and Classes
- Classes vs. modules in Python
file (name) - variables - functions - parameters - classes - elements - fields - methods - ==store/share== state between methods - multiple instances
import
參考
原生語法
- Pydon'ts | Mathspp
- Make Each Line Count, Keeping Things Simple In Python - Pybites
- all()
- List comprehensions
- set()
- from collections import Counter
- Every dunder method in Python - Python Morsels
- Object
- Python's Built-in Functions: A Complete Exploration – Real Python
- Python 單雙星號(* & **)解說 - MyApollo
- Python 淺談 with 語句 - MyApollo
- Python 模組教學 - 方便又實用的 dataclasses - MyApollo
- [Python] @contextmanager 裝飾器的使用方式 - Clay-Technology World
dictionary
sorted
copy - How to copy a dictionary and only edit the copy
get key - How to return dictionary keys as a list in Python?
```python= newdict = {1:0, 2:0, 3:0} [*newdict]
[1, 2, 3]
### list
[How do I concatenate two lists in Python?](https://stackoverflow.com/questions/1720421/how-do-i-concatenate-two-lists-in-python)
[Create list with numbers between 2 values](https://stackoverflow.com/questions/18265935/python-create-list-with-numbers-between-2-values?rq=1)
``` python=
list(range(11, 17))
# [11, 12, 13, 14, 15, 16]
dataclass
- Python3.7 特色(1)-dataclasses
- 淺談在Python中建立物件: class, namedtuple, NamedTuple and dataclass
- Data Classes vs typing.NamedTuple primary use cases
__str__ and __repr
Type hint
divmod
str
- What's the difference between str.isdigit, isnumeric and isdecimal in python
- Membership test operations
decorator
- dokelung.me
- 看了肯定會的 Python 裝飾子(decorator)教學 - MyApollo
- Python 复用装饰器代码 | 卡瓦邦噶!
- Python進階技巧 (3) — 神奇又美好的 Decorator ,嗷嗚! | by Jack Cheng | 整個程式都是我的咖啡館 | Medium
- real world example
案例
紀錄分層帶入的函式名稱和參數, 因為參數可能會在過程中被修改而造成錯誤 分層: view => business => dao
使用decorator 在過程中紀錄函式名稱和參數 名稱的部分使用qualname, 參數的部分使用args, kwargs 並在需要紀錄的函式加上該decorator
eval() and ast.literal_eval()
https://stackoverflow.com/questions/15197673/using-pythons-eval-vs-ast-literal-eval
ast.literal_eval() only considers a small subset of Python's syntax to be valid
Passing import('os').system('rm -rf /a-path-you-really-care-about') into ast.literal_eval() will raise an error, but eval() will happily delete your files.
re
- Regex Tester and Debugger Online - Javascript, PCRE, PHP
- python - What is the difference between re.search and re.match? - Stack Overflow
- re — Regular expression operations — Python 3.10.6 documentation
- py_regular_expressions
``` python= def get_previous_version(version: str, is_breaking_change: bool): previous_version = '1.0' if version is None: return previous_version matched = re.match('([\d]{1}).([\d]{1})', version) if matched: if is_breaking_change: previous_major_version = int(matched.group(1)) - 1 if int(matched.group(1)) > 0 else int(matched.group(1)) previous_version = f'{previous_major_version}.{matched.group(2)}' else: previous_minor_version = int(matched.group(2)) - 1 if int(matched.group(2)) > 0 else int(matched.group(2)) previous_version = f'{matched.group(1)}.{previous_minor_version}' return previous_version
``` python=
def is_feeback_type_valid(feedback_type: str) -> bool:
valid_length = len(feedback_type) <= 63
invalid_regulation = re.search('[^a-zA-Z0-9,_\s]{1,63}$', feedback_type)
return valid_length and not invalid_regulation
class name
```python= class Foo: def init(self): print(self.class.name)
```python=
import abc
class AService(abc.ABC):
pass
class A1Service(AService):
pass
a1 = A1Service()
print(A1Service.__class__.__name__)
print(A1Service.__name__)
print(a1.__class__.__name__)
print(a1.__name__)
# ABCMeta
# A1Service
# A1Service
# Traceback (most recent call last):
# File "name.py", line 14, in <module>
# print(a1.__name__)
# AttributeError: 'A1Service' object has no attribute '__name__'
global
- Python: change global variable from within another file
- Accessing module level variables, from within a function in the module
- sys.modules[name].var = something()
nonlocal
enum
``` python= class DoaminSource(Enum): A = (0, 'a') B = (1, 'b')
def __init__(self, identity, name):
self.id = identity
self.human_readable = name
def to_dict(self):
return {'id': self.id, 'name': self.human_readable}
@classmethod
def get_item_by_id(cls, source_id):
for enum_item in cls.__members__.values():
if source_id == enum_item.id:
return enum_item
```
isinstance
getattr
- Python using getattr to call function with variable parameters
- python - Access object methods from getattr() - Stack Overflow
- shell - Is it bad practice to use python's getattr extensively? - Stack Overflow
- what scenario can use the getattr and what is the security concern
functools
multiprocessing
- multiprocessing 模組進階篇 - Pipe, Queue, Array, RawArray 以及 Structure 之教學範例 - MyApollo
- Developing an Asynchronous Task Queue in Python | TestDriven.io
- Speeding Up Python with Concurrency, Parallelism, and asyncio | TestDriven.io
其他
lambda
Python Standard Library
logging
leverage log quickly
python=
FORMAT = '%(asctime)s %(levelname)s: %(message)s'
logging.basicConfig(level=logging.DEBUG, filename='filter.log', filemode='a', format=FORMAT)
logging.info('test')
``` python= import logging
logging.basicConfig( level=logging.DEBUG, format='[%(asctime)s][%(levelname)s][%(filename)s %(lineno)d]: %(message)s', handlers=[ logging.FileHandler('/tmp/foo.log'), logging.StreamHandler() ] ) logger = logging.getLogger(name)
- [Logging Cookbook](https://docs.python.org/3/howto/logging-cookbook.html#adding-contextual-information-to-your-logging-output)
- [Understanding logging in Python ](https://gist.github.com/mariocj89/73824162a3e35d50db8e758a42e39aab)
- [Understanding Python’s logging module](https://www.electricmonk.nl/log/2017/08/06/understanding-pythons-logging-module/)
- [Logging in Python like a PRO](https://guicommits.com/how-to-log-in-python-like-a-pro/)
- [Python logging: create log if not exists or open and continue logging if it does](https://stackoverflow.com/questions/41764941/python-logging-create-log-if-not-exists-or-open-and-continue-logging-if-it-does)
- [python - logger configuration to log to file and print to stdout - Stack Overflow](https://stackoverflow.com/questions/13733552/logger-configuration-to-log-to-file-and-print-to-stdout)
Loggers have a hierarchy
At the top of the hierarchy is the root logger, and the logger names don’t include the root logger
- created by asking a parent logger for a new child logger
- use dot notation
message propagation
the level of the message is **only checked** by the logger you give the message to. If the message’s level is lower or equal to the logger’s, the message is propagated up the hierarchy, but none of the other loggers will check the level! They’ll simply invoke their handlers.
level僅在進入點檢查一次
when giving a message to a logger, the logger checks the level. After that, the level on the loggers is no longer checked and all handlers in the entire chain are invoked, regardless of level.
When you log a message, the level is only checked at the logger you logged the message against. If it passes, every handler on every logger up the hierarchy is called, regardless of that logger’s level.
### time
- [Python 程式效能分析:量測運行時間 — 1010Code](https://andy6804tw.github.io//2023/06/22/time_profiler/)
ISO 8601
``` python=
from datetime import datetime, timezone, timedelta
datetime.now(timezone.utc).isoformat(timespec="seconds")
datetime.now(datetime.utcnow().astimezone().tzinfo).isoformat(timespec="seconds")
# iso 8601 string to datetime
result_time = datetime.datetime.strptime('2022-04-27T09:43:17+0800', '%Y-%m-%dT%H:%M:%S%z')
time shift ``` python= import datetime
expired = datetime.datetime.now() + datetime.timedelta(seconds=30) expired_timestamp = int(expired.timestamp())
str to datetime ``` python= import datetime d = datetime.datetime.strptime('2021-03-18 10:51:22', '%Y-%m-%d %H:%M:%S')
if the string is utc+0
d.astimezone(datetime.timezone.utc)
str to timestamp
``` python=
dt = datetime.datetime.strptime('2021-03-18 10:51:22', '%Y-%m-%d %H:%M:%S')
utc_time = dt.replace(tzinfo=datetime.timezone.utc)
utc_timestamp = utc_time.timestamp()
today format string ``` python= today = datetime.today().strftime('%Y%m%d')
### pathlib
- [Python pathlib 教學:檔案路徑操作超簡單,不再煩惱前斜線或後斜線! • 好豪筆記](https://haosquare.com/python-pathlib/)
- [Python 好用模組 - pathlib - MyApollo](https://myapollo.com.tw/blog/python-pathlib/)
make dir if not exist
``` python=
p_foo = Path('foo')
p_foo.mkdir(parents=True, exist_ok=True)
use exist Path ``` python= p_foo = Path('foo') p_csv = p_foo / 'csv'
find files by specific pattern
``` python=
p_foo = Path('/foo')
files = p_foo.glob('*.csv')
for file in files:
# handle the file
Get name of current script in Python ``` python= from pathlib import Path Path(file).name
csv
``` python=
with open(file) as csvfile: rows = csv.DictReader(csvfile) headers = list(rows.fieldnames) for row in rows: # logging.info(row)
### unittest
assertRaises
https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertRaises
``` python=
with self.assertRaises(SomeException) as cm:
do_something()
the_exception = cm.exception
self.assertEqual(the_exception.error_code, 3)
patch
shutil
- The Right Way to Run Shell Commands From Python | Martin Heinz | Personal Website & Blog
- Python subprocess 模組使用教學 - MyApollo
import subprocess
import shlex
ipmi_dcmi_power = shlex.split('ipmitool dcmi power reading')
ipmi_sdr_elist_full = shlex.split('ipmitool -c sdr elist full')
#p = subprocess.run('ipmitool dcmi power reading', shell=True, check=True, capture_output=True, encoding='utf-8')
p = subprocess.run(ipmi_dcmi_power, check=True, capture_output=True, encoding='utf-8')
print(f'Command {p.args} exited with {p.returncode} code, output: \n{p.stdout}')
for line in p.stdout.splitlines():
print(line.strip())
if 'Instantaneous power reading' in line.strip():
print(line.strip().split())
#p = subprocess.run('ipmitool -c sdr elist full', shell=True, check=True, capture_output=True, encoding='utf-8')
p = subprocess.run(ipmi_sdr_elist_full, check=True, capture_output=True, encoding='utf-8')
for line in p.stdout.splitlines():
print(line.strip())
print(line.strip().split(','))
``` python= import subprocess import logging import shlex
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s: %(message)s') logger = logging.getLogger(name)
def run(cmd_string): data = '' cmd = shlex.split(cmd_string) try: p = subprocess.run(cmd, check=True, capture_output=True, encoding='utf-8') logger.debug(f'Command {p.args} exited with {p.returncode} code, output: \n{p.stdout}') data = p.stdout except subprocess.SubprocessError as e: logger.error(e)
return data
``` python=
import asyncio
import logging
import datetime
import shlex
import threading
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s: %(message)s')
logger = logging.getLogger(__name__)
async def run(cmd, host, location):
logger.debug(f'{datetime.datetime.now().isoformat()}, thread id {threading.current_thread().ident}')
proc = await asyncio.create_subprocess_exec(
*shlex.split(cmd),
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE)
stdout, stderr = await proc.communicate()
logger.info(f'[{cmd!r} exited with {proc.returncode}]')
if stdout:
logger.debug(f'[stdout]\n{stdout.decode()}')
if stderr:
logger.debug(f'[stderr]\n{stderr.decode()}')
return host, location, proc.returncode, stdout.decode(), stderr.decode()
async def main(tasks):
logger.debug(f'main {datetime.now().isoformat()}, thread id {threading.current_thread().ident}')
return_values = await asyncio.gather(*tasks)
logger.debug(f'main {datetime.now().isoformat()}, thread id {threading.current_thread().ident}')
return return_values
def generate_tasks():
tasks = list()
tasks.append(run('cat abc', 'pd010', 'r1'))
tasks.append(run('cat abc', 'pd011', 'r2'))
if __name__ == '__main__':
tasks = generate_tasks()
return_values = asyncio.run(main(tasks))
for result in return_values:
logger.debug(f'result => {result}')
secrets
https://docs.python.org/3/library/secrets.html
```python=
random.randint(1, 1024)
import secrets secrets.choice(list(range(1, 1024)))### ipaddress
- [python - Convert an IP string to a number and vice versa - Stack Overflow](https://stackoverflow.com/questions/9590965/convert-an-ip-string-to-a-number-and-vice-versa)
### ayncio
- [Python asyncio 從不會到上路 | My.APOLLO](https://myapollo.com.tw/zh-tw/begin-to-asyncio/)
- [Asyncio Task Cancellation Best Practices - Super Fast Python](https://superfastpython.com/asyncio-task-cancellation-best-practices/)
- [python的asyncio模組(一):異步執行的好處 - iT 邦幫忙::一起幫忙解決難題,拯救 IT 人的一天](https://ithelp.ithome.com.tw/articles/10199385)
- [Strategies for Testing Async Code in Python](https://agariinc.medium.com/strategies-for-testing-async-code-in-python-c52163f2deab)
- [Testing asynchronous code in Python](https://promity.com/2020/06/03/testing-asynchronous-code-in-python/)
- subprocess
- [python 3.x - Using asyncio to wait for results from subprocess - Stack Overflow](https://stackoverflow.com/questions/63782892/using-asyncio-to-wait-for-results-from-subprocess)
- [Subprocesses — Python 3.11.5 documentation](https://docs.python.org/3/library/asyncio-subprocess.html)
### sqllite
- [Recipes from Python SQLite docs · Redowan's Reflections](https://rednafi.github.io/reflections/recipes-from-python-sqlite-docs.html)
### fcntl
- [Python - fcntl.flock(fd, op) 檔案鎖應用範例 - MyApollo](https://myapollo.com.tw/blog/python-fcntl-flock/)
- [Python : Locking text file on NFS - Stack Overflow](https://stackoverflow.com/questions/37633951/python-locking-text-file-on-nfs)
- [Barry Warsaw / FLUFL Lock · GitLab](https://gitlab.com/warsaw/flufl.lock)
### xml
- [xml.etree.ElementTree 筆記 - HackMD](https://hackmd.io/@top30339/rJYlKYpml?type=view)
- [Updating XML elements and attribute values using Python etree - Stack Overflow](https://stackoverflow.com/questions/9177360/updating-xml-elements-and-attribute-values-using-python-etree)
- [xml - Python Element Tree Writing to New File - Stack Overflow](https://stackoverflow.com/questions/37713184/python-element-tree-writing-to-new-file)
- [python - How to write XML declaration using xml.etree.ElementTree - Stack Overflow](https://stackoverflow.com/questions/15356641/how-to-write-xml-declaration-using-xml-etree-elementtree)
``` python
import sys
import logging
import xml.etree.ElementTree as ET
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s: %(message)s')
logger = logging.getLogger(__name__)
if __name__ == '__main__':
try:
benchmark_bios_config = sys.argv[1]
logger.info(f'benchmark_bios_config: {benchmark_bios_config}')
tree = ET.parse(benchmark_bios_config)
root = tree.getroot()
settings = root.findall(".//Setting")
for setting in settings:
logger.info(f"{setting.get('name')} {setting.get('selectedOption')}")
default_setting = root.find(f".//Setting[@name='foo']")
default_setting.set('selectedOption', 'bar')
with open('output_file', 'wb') as f:
tree.write(f, encoding='utf-8')
except IndexError as e:
logger.error(e)
except ET.ParseError as e:
logger.error(e)
difflib
nice tools
- Python's Missing Batteries: Essential Libraries You're Missing Out On | Martin Heinz | Personal Website & Blog
- Python for DevOps | Pulumi Blog
configparser
CLI彈性輸入
command-line argument parsing library
- configparser
- click
- typer
- prompt-toolkit/python-prompt-toolkit: Library for building powerful interactive command line applications in Python
ssh tunnel
- https://blog.ruanbekker.com/blog/2018/04/23/setup-a-ssh-tunnel-with-the-sshtunnel-module-in-python/
streaming large json
- ijson
- bigjson
logger
face_recognition
Python dependency visualization
- Import structure
- Classes hierachy
- Pyreverse in pylint
- analyses Python code and extracts UML class diagrams and package dependencies
- 使用Graphviz和Pyreverse绘制Python项目结构图 | 夜不能寐
- Pyreverse in pylint
- visual scripting framework for python
Faker
project template
change log
pandas
- How do I read and write tabular data? — pandas 2.1.1 documentation
- Comparison with SQL — pandas 2.1.1 documentation
- pandas.DataFrame.between_time — pandas 2.1.1 documentation
- Pandas 中如何從文字檔案載入資料 | D棧 - Delft Stack
- python - Dataframe set_index not setting - Stack Overflow
- Python Pandas Select Index where index is larger than x - Stack Overflow
read multiple csv https://stackoverflow.com/questions/20906474/import-multiple-csv-files-into-pandas-and-concatenate-into-one-dataframe
```python= import glob
df = pd.concat(map(pd.read_csv, glob.glob('data/*.csv')))
example
``` python
import os
import glob
import pandas as pd
date = '11-08'
begin_time = '07:47'
end_time = '08:45'
manual_gpu = 6000
# single file
#df = pd.read_table("/opt/power_consumption/data/pdu/d23pd015_r9v-l_snmpwalk.txt", sep='\s+')
# multiple files
path = '/opt/power_consumption/data/pdu'
all_files = glob.glob(os.path.join(path, "*.txt"))
df = pd.concat((pd.read_table(f,sep='\s+') for f in all_files), ignore_index=True)
df["time"] = pd.to_datetime(df['TimeStamp'],format= '%m%d:%H%M' )
df.set_index('time', inplace=True)
df['all-w'] = df['1-w'] + df['2-w'] + df['3-w']
print(df)
df.to_csv('all_pdu.csv', index=False)
#print(df[df.index >= pd.to_datetime(date, format='%m-%d')].between_time(begin_time, end_time))
#print(df[df.index >= pd.to_datetime(date, format='%m-%d')].between_time(begin_time, end_time).groupby("TimeStamp").agg({'all-w': 'sum'}))
#print(df[(df.index >= pd.to_datetime(date, format='%m-%d')) & (df['TimeStamp'] == '1108:0752' )])
#pdu_df = df[df.index >= pd.to_datetime(date, format='%m-%d')].between_time(begin_time, end_time).groupby(["Host", "TimeStamp"]).agg({'all-w': 'sum'}).reset_index()
#print(pdu_df)
#print(pdu_df[pdu_df['TimeStamp'] == '1108:0752'])
final_df = df[df.index >= pd.to_datetime(date, format='%m-%d')].between_time(begin_time, end_time).groupby("TimeStamp").agg({'all-w': 'sum'})
final_df['all-w-with-gpu'] = final_df['all-w'] + manual_gpu
print(final_df)
print(final_df.describe())
MySQL
- What's the difference between PyMySQL and mysqlclient-python
- What's the difference between MySQLdb, mysqlclient and MySQL connector/Python
ORM
engine session base
SQLAlchemy
- Unlocking the Power of Composite Primary Keys in SQLAlchemy | by Thinesh Ramanathan | Medium
- python - SQLAlchemy ORM not working with composite foreign keys - Stack Overflow
- Postgresql + Sqlalchemy ORM CRUD方式 - HackMD
database migration
- 使用 Alembic 來進行資料庫版本管理
- Get Started With Alembic
- FastAPI with Async SQLAlchemy, SQLModel, and Alembic
- Sqlalchemy with Flask-Migrate - HackMD
secure
PyCryptodome
signature with PSS https://pycryptodome.readthedocs.io/en/latest/src/signature/pkcs1_pss.html ``` python= from Crypto.Signature import pss from Crypto.Hash import SHA256 from Crypto.PublicKey import RSA from Crypto import Random from base64 import b64encode, b64decode
message = 'foo'.encode() key = RSA.import_key(open('private_key.pem').read()) h = SHA256.new(message) signature = pss.new(key).sign(h) sig_base64 = b64encode(signature).decode() print(sig_base64)
key = RSA.import_key(open('public_key.pem').read()) a = SHA256.new(message) verifier = pss.new(key) try: verifier.verify(a, b64decode(sig_base64)) print("The signature is authentic.") except (ValueError, TypeError): print("The signature is not authentic.")
### passlib
- [How to generate a password hash in Python? - The Security Buddy](https://www.thesecuritybuddy.com/cryptography-and-python/how-to-generate-a-password-hash-in-python/)
### requests
- [Advanced usage of Python requests - timeouts, retries, hooks](https://findwork.dev/blog/advanced-usage-python-requests-timeouts-retries-hooks/)
- [Requests Timeouts](https://docs.python-requests.org/en/master/user/advanced/#timeouts)
- [Send HTTP requests using python-requests with timeout, tcp reuse(session) and retry. · GitHub](https://gist.github.com/laixintao/e9eae48a835e741969ae06af3ad45f71)
- https://twitter.com/laixintao/status/1737693034270339406
Request hooks and `raise_for_status()`
timeouts
Retry on failure
``` python=
from http import HTTPStatus
try:
...
r = requests.post()
r.raise_for_status()
except requests.exceptions.HTTPError as err:
if r.status_code == HTTPStatus.TOO_MANY_REQUESTS:
pass
pyppeteer
- Read html file and make pdf with pyppeteer · GitHub
- python - Is it possible to get pdf page using pyppeteer? - Stack Overflow
- GitHub - kiwi0fruit/pyppdf: Pyppeteer PDF. Print html sites and files to pdf via pyppeteer (uses patched pyppeteer that by default downloads updated Chromium revision via https with certifi).
ruff
- GitHub - astral-sh/ruff: An extremely fast Python linter and code formatter, written in Rust.
-
Ruff 是由 Rust 編寫的高速 Python 檢查器和程式碼格式化工具,可通過 pip 安裝,支援至 Python 3.12。它具有超過 700 個內建規則,並能替代 Flake8、isort 和 Black 等工具,且執行速度遠超過這些工具。https://twitter.com/daininduyuanma/status/1717141469822386435
-
- 新世代的 Python Linter - Ruff - MyApollo
- Python 開發:Ruff Linter、Formatter 介紹 + 設定教學 - Code and Me
csvtotable
fingerprint
Type helper
pydantic 最大的問題是它也有所謂的 model 和 schema,當再把資料庫的概念摻進來就造成語意上的混亂。 不僅是語意上,程式碼看起來也會相當冗餘,以前面的 User 資源為例,除了有 pydantic model 的定義外,在 ORM 大概也要寫一份極其相似的「model」
web framework
malware
markdown
sqlite utils
iwhat
ipmi
def print_sdr_list_entry(record_id, number, id_string, value, states):
if number:
number = str(number)
else:
number = 'na'
if states:
states = hex(states)
else:
states = 'na'
print("0x%04x | %3s | %-18s | %9s | %s" % (record_id, number,
id_string, value, states))
# Example usage
ip = "167.222.1.33"
username = "foo"
password = "foo"
interface = pyipmi.interfaces.create_interface(interface='rmcp',
slave_address=0x81,
host_target_address=0x20,
keep_alive_interval=1)
ipmi = pyipmi.create_connection(interface)
ipmi.session.set_session_type_rmcp(host=ip, port=623)
ipmi.session.set_auth_type_user(username=username, password=password)
ipmi.target = pyipmi.Target(ipmb_address=0x20)
ipmi.session.establish()
device_id = ipmi.get_device_id()
print('''
Device ID: %(device_id)s
Device Revision: %(revision)s
Firmware Revision: %(fw_revision)s
IPMI Version: %(ipmi_version)s
Manufacturer ID: %(manufacturer_id)d (0x%(manufacturer_id)04x)
Product ID: %(product_id)d (0x%(product_id)04x)
Device Available: %(available)d
Provides SDRs: %(provides_sdrs)d
Additional Device Support:
'''[1:-1] % device_id.__dict__)
functions = (
('SENSOR', 'Sensor Device'),
('SDR_REPOSITORY', 'SDR Repository Device'),
('SEL', 'SEL Device'),
('FRU_INVENTORY', 'FRU Inventory Device'),
('IPMB_EVENT_RECEIVER', 'IPMB Event Receiver'),
('IPMB_EVENT_GENERATOR', 'IPMB Event Generator'),
('BRIDGE', 'Bridge'),
('CHASSIS', 'Chassis Device')
)
for n, s in functions:
if device_id.supports_function(n):
print(' %s' % s)
if device_id.aux is not None:
print('Aux Firmware Rev Info: [%s]' % (
' '.join('0x%02x' % d for d in device_id.aux)))
for selector in range(1, 6):
caps = ipmi.get_dcmi_capabilities(selector)
print('Selector: {} '.format(selector))
print(' version: {} '.format(caps.specification_conformence))
print(' revision: {}'.format(caps.parameter_revision))
print(' data: {}'.format(caps.parameter_data))
rsp = ipmi.get_power_reading(1)
print('Power Reading')
print(' current: {}'.format(rsp.current_power))
print(' minimum: {}'.format(rsp.minimum_power))
print(' maximum: {}'.format(rsp.maximum_power))
print(' average: {}'.format(rsp.average_power))
print(' timestamp: {}'.format(rsp.timestamp))
print(' period: {}'.format(rsp.period))
print(' state: {}'.format(rsp.reading_state))
for s in ipmi.device_sdr_entries():
try:
number = None
value = None
states = None
if s.type is pyipmi.sdr.SDR_TYPE_FULL_SENSOR_RECORD:
(value, states) = ipmi.get_sensor_reading(s.number)
number = s.number
if value is not None:
value = s.convert_sensor_raw_to_value(value)
elif s.type is pyipmi.sdr.SDR_TYPE_COMPACT_SENSOR_RECORD:
(value, states) = ipmi.get_sensor_reading(s.number)
number = s.number
id_string = getattr(s, 'device_id_string', None)
print_sdr_list_entry(s.id, number, id_string, value, states)
except pyipmi.errors.CompletionCodeError as e:
if s.type in (pyipmi.sdr.SDR_TYPE_COMPACT_SENSOR_RECORD,
pyipmi.sdr.SDR_TYPE_FULL_SENSOR_RECORD):
print('0x{:04x} | {:3d} | {:18s} | ERR: CC=0x{:02x}'.format(
s.id,
s.number,
s.device_id_string,
e.cc))
Streamlit
- Streamsync — Streamlit 的挑戰者 - MyApollo
- (2023年新版) 使用 Streamlit 簡單又快速建立 Dashboard 網頁 – IT 空間
- Streamlit cheat sheet · Streamlit
- App Gallery • Streamlit
Celery
pytest
mkdocs
- Deploy MkDocs · Actions · GitHub Marketplace · GitHub
- Document your project with MkDocs and GitHub Pages - DEV Community
use the system daemon /etc/systemd/system/xxx.service
[Unit]
Description=Job that runs the python SimpleHTTPServer daemon
Documentation=man:SimpleHTTPServer(1)
[Service]
Type=simple
WorkingDirectory=/home/supermicro/smci-doc/site
ExecStart=/usr/bin/python3 -m http.server 8888
ExecStop=/bin/kill `/bin/ps aux | /bin/grep http.server | /bin/grep -v grep | /usr/bin/awk '{ print $2 }'`
[Install]
WantedBy=multi-user.target
site_name: My Docs
# plugins:
# - offline
markdown_extensions:
- admonition
- pymdownx.details
- pymdownx.highlight:
anchor_linenums: true
line_spans: __span
pygments_lang_class: true
- pymdownx.inlinehilite
- pymdownx.snippets
- pymdownx.superfences
- pymdownx.tabbed:
alternate_style: true
theme:
name: material
features:
- navigation.top
- navigation.path
- search.highlight
- search.share
- navigation.footer
- content.code.copy
- content.code.select
- content.code.annotate
extra:
social:
- icon: fontawesome/brands/gitlab
link: https://gitlab.supermicro.com/
copyright: Copyright © 2023
pywifi
pyee
diagrams
airflow
ansible-playbook-grapher
glances
tw_stocker
Docling
alternative: microsoft/markitdown: Python tool for converting files and office documents to Markdown.
garmin-grafana
software development tools
- Python的一些套件(black、isort、pylint) + pre-commit - HackMD
- pre-commit
- git-changelog
.pre-commit-config.yaml