Skip to content

Instantly share code, notes, and snippets.

@lgray

lgray/spark_works_again.txt

Created December 9, 2022 22:47
Show Gist options
  • Save lgray/17c182e20a6cf3b78b0c7c35a1cb9400 to your computer and use it in GitHub Desktop.
Save lgray/17c182e20a6cf3b78b0c7c35a1cb9400 to your computer and use it in GitHub Desktop.
Download ZIP
nanoaod in spark table
Raw
spark_works_again.txt
(coffea-dev) lgray@dhcp-131-225-97-134 coffea % python -i spark_work.py
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/12/09 16:42:22 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
40 * {Muon_pt: var * float32, Muon_eta: var * float32, Muon_phi: var * float32, Muon_mass: var * float32, Muon_charge: var * int32, nMuon: int64}
[pyarrow.RecordBatch
Muon_pt: list<item: float not null> not null
child 0, item: float not null
Muon_eta: list<item: float not null> not null
child 0, item: float not null
Muon_phi: list<item: float not null> not null
child 0, item: float not null
Muon_mass: list<item: float not null> not null
child 0, item: float not null
Muon_charge: list<item: int32 not null> not null
child 0, item: int32 not null
nMuon: int64 not null]
+--------------------+--------------------+--------------------+--------------------+-----------+-----+
| Muon_pt| Muon_eta| Muon_phi| Muon_mass|Muon_charge|nMuon|
+--------------------+--------------------+--------------------+--------------------+-----------+-----+
| []| []| []| []| []| 0|
| []| []| []| []| []| 0|
| []| []| []| []| []| 0|
| []| []| []| []| []| 0|
|[76.75332, 20.131...|[0.713501, 2.3623...|[1.6398926, -2.87...|[0.10571289, 0.10...| [-1, 1]| 2|
| [31.038704]| [-0.62109375]| [-1.3996582]| [0.10571289]| [-1]| 1|
|[50.641342, 14.33...|[0.09056091, -2.0...|[0.47424316, -2.6...|[0.10571289, 0.10...| [-1, 1]| 2|
| []| []| []| []| []| 0|
| []| []| []| []| []| 0|
| []| []| []| []| []| 0|
| [16.724983]| [0.6816406]| [-3.034668]| [0.10571289]| [1]| 1|
| []| []| []| []| []| 0|
| []| []| []| []| []| 0|
| []| []| []| []| []| 0|
| [13.908063]| [1.0]| [-0.9782715]| [0.10571289]| [1]| 1|
| []| []| []| []| []| 0|
| []| []| []| []| []| 0|
| []| []| []| []| []| 0|
| []| []| []| []| []| 0|
| []| []| []| []| []| 0|
+--------------------+--------------------+--------------------+--------------------+-----------+-----+
only showing top 20 rows
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment