DICOM writer: initial support for dual personality DICOM/TIFF #3911

melissalinkert · 2022-11-18T20:14:25Z

The DICOM writer now includes TIFF metadata as described in https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6489422/.

This requires more testing before it is ready for final review, but I would expect basic test cases to produce files that can be read as either DICOM or TIFF. For example, using a public brightfield slide (https://downloads.openmicroscopy.org/images/Vectra-QPTIFF/perkinelmer/PKI_scans/HandEcompressed_Scan1.qptiff):

$ bfconvert ~/data/bf-data-repo/automated-tests/curated/vectra-qptiff/public/perkinelmer/PKI_scans/HandEcompressed_Scan1.qptiff qptiff-test.dcm -noflat -tilex 512 -tiley 512

tiffinfo and/or tiffdump should be able to read the output qptiff-test*.dcm files. showinf -format MinimalTiff on any of the output files forces Bio-Formats to read as TIFF, and should display an image of the correct size. For the larger resolutions, it may be necessary to use the -crop x,y,w,h option to read a smaller region from the full image. As discussed with @fedorov, @dclunie, and @erindiel, only basic TIFF metadata is written. There is no OME-XML written, so these won't be recognized as OME-TIFF (much less pyramid OME-TIFF). As such, when reading as TIFF each file will be detected individually and the pyramid/label/macro/etc. relationships will not be visible.

showinf -noflat qptiff-test_0_0.dcm should continue to detect the whole DICOM dataset, just like DICOM data written without this pull request. dcdump/dciodvfy should also be able to read the output files as before.

By default, non-BigTIFF files are written, unless the total pixel count approaches 4GB. The -bigtiff option to showinf will force BigTIFF metadata to be written instead.

I have not yet extensively tested different compression options, different tile sizes, or a wider range of input data. I would expect some fine-tuning to be needed here as testing continues.

Fixes #3856 as noted in #3911 (review)

...so that other writers can use it too.

…fset Useful for cases where all IFDs are written sequentially.

joshmoore · 2022-11-21T08:57:13Z

As discussed with @fedorov, @dclunie, and @erindiel, only basic TIFF metadata is written. There is no OME-XML written, so these won't be recognized as OME-TIFF (much less pyramid OME-TIFF). As such, when reading as TIFF each file will be detected individually and the pyramid/label/macro/etc. relationships will not be visible.

Could more of the background be added here? Is this more a design, peformance, or just time constraint?

melissalinkert · 2022-11-21T16:22:32Z

1572bae and 0e65225 are the commits from #3912, cherry-picked onto this branch so that we can have pre-built jars here to help with review. The current build artifact is: ~~https://github.com/melissalinkert/bioformats/suites/9425153710/artifacts/445347374~~ https://github.com/melissalinkert/bioformats/suites/9426389714/artifacts/445435641. The artifacts are accessible by clicking the ✔️ next to the latest commit, clicking any of the Details links, clicking the Summary link in the left-hand panel, and then scrolling to the Artifacts section at the bottom. The artifact.zip should contain:

 bfmatlab.zip            
 bftools.zip             
 bio-formats-testing-framework.jar  
 bio-formats-tools.jar   
 bio-formats_plugins.jar  
 bioformats-octave-6.11.1-SNAPSHOT.tar.gz  
 bioformats_package.jar  
 formats-api.jar         
 formats-bsd.jar         
 formats-gpl.jar         
 turbojpeg.jar

@joshmoore : leaving out the OME-XML is mostly a time constraint at the moment. We may add it later, but want to make sure that the basic TIFF metadata writing is working first.

joshmoore · 2022-11-21T16:43:07Z

@joshmoore : leaving out the OME-XML is mostly a time constraint at the moment. We may add it later, but want to make sure that the basic TIFF metadata writing is working first.

👍

Setting the "dicom.dual_personality" option to false, e.g. $ bfconvert -option dicom.dual_personality false /path/to/input /path/to/output.dcm will omit the TIFF header from the preamble, as well as the trailing padding tag that would have contained IFDs. By default, dual personality data will still be written.

melissalinkert · 2022-11-29T22:45:03Z

@dclunie @fedorov : the latest commit here (a386b0a) adds an option to turn off dual personality writing. Setting the dicom.dual_personality option to false as described in https://docs.openmicroscopy.org/bio-formats/6.11.0/formats/options.html#writer-options should write "plain" DICOM files. Pre-built jars that include that commit are here:

~~https://github.com/ome/bioformats/suites/9573896272/artifacts/455949858~~
https://github.com/ome/bioformats/suites/10428627349/artifacts/515643112

…icom-dual-personality

JPEG still needs some work to be compliant with ImageMagick.

See ome#3856

melissalinkert · 2023-01-26T22:22:39Z

As noted in the commit message for 63d75e0, #3856 is related in the case where DICOM with a TIFF header is written with JPEG-compressed pixel data. I'd expect 63d75e0 to be the minimal set of changes needed to fix that issue.

…icom-dual-personality

melissalinkert · 2023-06-16T20:25:41Z

Merge conflicts resolved, exclude label removed in preparation for review. There are some very minor DicomReader changes here, but I don't expect that to impact tests; I'd expect all builds to remain green.

In addition to what's noted above, I'd expect the following:

bfconvert -noflat -option dicom.dual_personality false /path/to/input /path/to/output.dcm results in DICOM files that do not contain TIFF metadata. showinf -format MinimalTiff /path/to/output.dcm and/or tiffinfo can verify. showinf -noflat /path/to/output.dcm should correctly read as DICOM.
bfconvert -noflat /path/to/input /path/to/output.dcm results in DICOM files that do contain TIFF metadata. showinf -noflat /path/to/output.dcm should read as DICOM. showinf -nopix -format MinimalTiff /path/to/output.dcm should initialize; as noted above, the TIFF metadata is minimal, so the whole pyramid will not be detected.
bfconvert -noflat -compression JPEG ... and bfconvert -noflat -compression JPEG-2000 ... should produce JPEG/JPEG-2000 compressed data that is readable as both DICOM and TIFF. Note other compression types are not supported (e.g. LZW).
Other uses of bfconvert should not be impacted, as the only change here is to pass the -bigtiff option to the DICOM writer.

This is best tested with brightfield and fluorescence whole slides, but fake data could be used as an input as well.

dclunie · 2023-06-18T14:06:43Z

I tested:

./bfconvert -noflat -tilex 256 -tiley 256 -compression JPEG CMU-1.svs /tmp/wsiconverteddual/crap.dcm

and it works fine, but adding the "-option dicom.dual_personality false" failed ...

./bfconvert -noflat -option dicom.dual_personality false -tilex 256 -tiley 256 -compression JPEG CMU-1.svs /tmp/wsiconverteddual/crap.dcm
CMU-1.svs
SVSReader initializing CMU-1.svs
Reading IFDs
Populating metadata
Populating OME metadata
[Aperio SVS] -> /tmp/wsiconverteddual/crap.dcm [DICOM]
Tile size = 256 x 256
Exception in thread "main" java.lang.NullPointerException
	at loci.formats.out.DicomWriter.saveBytes(DicomWriter.java:212)
	at loci.formats.ImageWriter.saveBytes(ImageWriter.java:260)
	at loci.formats.tools.ImageConverter.convertTilePlane(ImageConverter.java:970)
	at loci.formats.tools.ImageConverter.convertPlane(ImageConverter.java:833)
	at loci.formats.tools.ImageConverter.testConvert(ImageConverter.java:766)
	at loci.formats.tools.ImageConverter.main(ImageConverter.java:1199)

As noted in the comment, the IFDs will always be null if dual personality writing is turned off.

melissalinkert · 2023-06-20T01:34:15Z

adding the "-option dicom.dual_personality false" failed ..

Thanks, @dclunie. This should be fixed with the changes in 0583e88.

melissalinkert · 2023-06-30T16:44:38Z

Thanks, @sbesson. That's an edge case with files that have more than 2GB of pixel data. Notably, bfconvert HandEcompressed_Scan1.qptiff qptiff-test.dcm -noflat -tilex 512 -tiley 512 -compression JPEG (just adding compression) worked as expected.

d56bcdd should fix this; I believe it was only a reader issue and not a writer issue.

dgault · 2023-06-30T17:57:20Z

Seeing a similar, but slightly different exception when converting the ndpi file DM0014 - 2020-04-02 11.10.47.ndpi

With the new option set to false, tile size was 512, attempting to read the individual resolutions with the default DicomReader results in the below exception (the exception below was taken from the lowest resolution level):

Exception in thread "main" java.io.EOFException: Attempting to read beyond end of file.
	at loci.common.NIOFileHandle.readShort(NIOFileHandle.java:452)
	at loci.common.RandomAccessInputStream.readShort(RandomAccessInputStream.java:646)
	at loci.formats.dicom.DicomTag.<init>(DicomTag.java:274)
	at loci.formats.dicom.DicomTag.<init>(DicomTag.java:90)
	at loci.formats.in.DicomReader.initFile(DicomReader.java:441)

Moving each file to a seperate folder and each then is able to be opened individually without the exception.

dgault · 2023-06-30T18:04:56Z

Sorry, missed the most recent commit, initial retest looks promising

sbesson

Retested with the last commit. Using the sample image, generated a series of DICOM WSI files using all supported compression schemes

./artifact/bftools/bfconvert HandEcompressed_Scan1.qptiff  default.dcm -tilex 512 -tiley 512 -noflat  
./artifact/bftools/bfconvert HandEcompressed_Scan1.qptiff jpeg.dcm -tilex 512 -tiley 512 -noflat -compression JPEG
./artifact/bftools/bfconvert HandEcompressed_Scan1.qptiff j2k.dcm -tilex 512 -tiley 512 -noflat -compression JPEG-2000
./artifact/bftools/bfconvert HandEcompressed_Scan1.qptiff nodual.dcm -tilex 512 -tiley 512 -noflat -option dicom.dual_personality false

Confirmed that for all samples except the one where the dual personality flag is set to false, the generated files are recognized as TIFF with the appropriate compression scheme

(java11) sbesson@Sebastiens-MacBook-Pro-2 Downloads % for i in $(ls *.dcm); do echo $i && tiffinfo $i | grep Compression; done
default_0_0.dcm
  Compression Scheme: None
default_0_1.dcm
  Compression Scheme: None
default_0_2.dcm
  Compression Scheme: None
default_0_3.dcm
  Compression Scheme: None
default_0_4.dcm
  Compression Scheme: None
default_1_0.dcm
  Compression Scheme: None
default_2_0.dcm
default_2_0.dcm: Warning, Nonstandard tile width 2084, convert file.
default_2_0.dcm: Warning, Nonstandard tile length 4024, convert file.
  Compression Scheme: None
default_3_0.dcm
default_3_0.dcm: Warning, Nonstandard tile width 521, convert file.
default_3_0.dcm: Warning, Nonstandard tile length 449, convert file.
  Compression Scheme: None
j2k_0_0.dcm
  Compression Scheme: 33003 (0x80eb)
j2k_0_1.dcm
  Compression Scheme: 33003 (0x80eb)
j2k_0_2.dcm
  Compression Scheme: 33003 (0x80eb)
j2k_0_3.dcm
  Compression Scheme: 33003 (0x80eb)
j2k_0_4.dcm
  Compression Scheme: 33003 (0x80eb)
j2k_1_0.dcm
  Compression Scheme: 33003 (0x80eb)
j2k_2_0.dcm
j2k_2_0.dcm: Warning, Nonstandard tile width 2084, convert file.
j2k_2_0.dcm: Warning, Nonstandard tile length 4024, convert file.
  Compression Scheme: 33003 (0x80eb)
j2k_3_0.dcm
j2k_3_0.dcm: Warning, Nonstandard tile width 521, convert file.
j2k_3_0.dcm: Warning, Nonstandard tile length 449, convert file.
  Compression Scheme: 33003 (0x80eb)
jpeg_0_0.dcm
  Compression Scheme: JPEG
jpeg_0_1.dcm
  Compression Scheme: JPEG
jpeg_0_2.dcm
  Compression Scheme: JPEG
jpeg_0_3.dcm
  Compression Scheme: JPEG
jpeg_0_4.dcm
  Compression Scheme: JPEG
jpeg_1_0.dcm
  Compression Scheme: JPEG
jpeg_2_0.dcm
jpeg_2_0.dcm: Warning, Nonstandard tile width 2084, convert file.
jpeg_2_0.dcm: Warning, Nonstandard tile length 4024, convert file.
  Compression Scheme: JPEG
jpeg_3_0.dcm
jpeg_3_0.dcm: Warning, Nonstandard tile width 521, convert file.
jpeg_3_0.dcm: Warning, Nonstandard tile length 449, convert file.
  Compression Scheme: JPEG
nodual_0_0.dcm
nodual_0_0.dcm: Not a TIFF or MDI file, bad magic number 0 (0x0).
nodual_0_1.dcm
nodual_0_1.dcm: Not a TIFF or MDI file, bad magic number 0 (0x0).
nodual_0_2.dcm
nodual_0_2.dcm: Not a TIFF or MDI file, bad magic number 0 (0x0).
nodual_0_3.dcm
nodual_0_3.dcm: Not a TIFF or MDI file, bad magic number 0 (0x0).
nodual_0_4.dcm
nodual_0_4.dcm: Not a TIFF or MDI file, bad magic number 0 (0x0).
nodual_1_0.dcm
nodual_1_0.dcm: Not a TIFF or MDI file, bad magic number 0 (0x0).
nodual_2_0.dcm
nodual_2_0.dcm: Not a TIFF or MDI file, bad magic number 0 (0x0).
nodual_3_0.dcm
nodual_3_0.dcm: Not a TIFF or MDI file, bad magic number 0 (0x0).

Also compared the reading of a selected region of the largest resolution

./artifact/bftools/showinf -format MinimalTiff jpeg_0_0.dcm -crop 1024,1024,2048,2048
./artifact/bftools/showinf -format MinimalTiff jpeg_0_0.dcm -crop 10240,10240,2048,2048
./artifact/bftools/showinf -format MinimalTiff j2k_0_0.dcm -crop 10240,10240,2048,2048
./artifact/bftools/showinf -format MinimalTiff nodual_0_0.dcm -crop 10240,10240,2048,2048
./artifact/bftools/showinf -format MinimalTiff default_0_0.dcm -crop 10240,10240,2048,2048
./artifact/bftools/showinf default_0_0.dcm -crop 10240,10240,2048,2048
./artifact/bftools/showinf nodual_0_0.dcm -crop 10240,10240,2048,2048 -noflat

as well as the reading of a low resolution

./artifact/bftools/showinf nodual_0_0.dcm -noflat -resolution 4
./artifact/bftools/showinf default_0_0.dcm -noflat -resolution 4
./artifact/bftools/showinf -format MinimalTiff default_0_4.dcm 
./artifact/bftools/showinf -format MinimalTiff jpeg_0_4.dcm 
./artifact/bftools/showinf -format MinimalTiff j2k_0_4.dcm

All the results are visually consistent with each other. Based on the above, this looks ready for inclusion in the upcoming release of Bio-Formats

dgault · 2023-07-03T13:40:25Z

Retesting and still seeing some similar issues:

Using sample file https://downloads.openmicroscopy.org/images/Hamamatsu-NDPI/openslide/CMU-3/
bfconvert /hamamatsu/public/openslide/CMU-3/CMU-3.ndpi CMU-3.dcm -noflat -tilex 512 -tiley 512 -option dicom.dual_personality false

Same issue with or without the dual personality flag and seeing the same thing with different Java versions.
Reading using showinf, default options:

Exception in thread "main" java.lang.NegativeArraySizeException: -992887868
	at loci.formats.dicom.DicomTag.<init>(DicomTag.java:201)
	at loci.formats.dicom.DicomTag.<init>(DicomTag.java:90)
	at loci.formats.in.DicomReader.initFile(DicomReader.java:439)

Also seeing failures in the repo tests this morning: https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-test-folder/lastUnsuccessfulBuild/console

melissalinkert · 2023-07-03T13:48:43Z

Excluding and bumping to 7.0.0 as discussed earlier, I'll pick this back up next week.

melissalinkert · 2023-07-12T21:48:49Z

Removing exclude label, as I believe the last few commits here should fix the tests.

The issue noted in #3911 (comment) is addressed in c891f10; that will fail fast in the case where more than 4GB of pixel data would be written to a single file, with no compression. To my knowledge, there is no valid way to write more than 4GB of uncompressed pixel bytes (see e.g. the Pixel Matrix section of https://dicom.nema.org/dicom/dicomwsi/#Overview). Specifying a compression type (as most users would do) should allow larger slides to be written.

...since compression isn't set until after setId.

melissalinkert · 2023-07-13T23:13:49Z

Assuming builds pass overnight, I have no more planned changes here. I have retested conversions listed in:

with 0f7d1c6 and no longer see issues with converting or reading the results. I have also run most of the automated tests locally with 0f7d1c6 and not observed any failures so far.

dgault

Retested using the sample sets that had failed previously:
https://downloads.openmicroscopy.org/images/Vectra-QPTIFF/perkinelmer/PKI_scans/HandEcompressed_Scan1.qptiff
https://downloads.openmicroscopy.org/images/Hamamatsu-NDPI/openslide/CMU-3/
https://downloads.openmicroscopy.org/images/Hamamatsu-NDPI/hamamatsu/DM0014%20-%202020-04-02%2011.10.47.ndpi

Tested using bfconvert, with and without the dual personality flag. When the flag is set to false the resulting DICOM files do not contain any TIFF metadata whne tested with tiffinfo. The files are still able to opened and read as standard DICOM as expected.

When the flag is set or true (or by default without the flaf set), then the resulting DICOM files are able to recognised as TIFF and tiffinfo displays the associated metadata. The files are able to be opened and displayed as expected both by default and when using the -format flag in bfconvert to force reading with MinimalTiffReader.

Retested the same files, this time using compression set to JPEG and JPEG-2000. In both instances the resulting files are able to read and opened, either as DICOM or TIFF.

Overall the PR looks good to merge from my side.

sbesson

Retested using two of the source samples mentioned above with the following conversion options:

Source	Tile size	Compression	Dual-personality
CMU-3.npdi	512x512	None	Yes
CMU-3.npdi	2048x2048	None	Yes
CMU-3.npdi	512x512	JPEG-2000	Yes
CMU-3.npdi	512x512	JPEG	Yes
CMU-3.npdi	512x512	None	No
CMU-3.npdi	512x512	JPEG	No
HandEcompressed_Scan1.qptiff	512x512	None	Yes
HandEcompressed_Scan1.qptiff	2048x2048	None	Yes
HandEcompressed_Scan1.qptiff	512x512	JPEG-2000	Yes
HandEcompressed_Scan1.qptiff	512x512	JPEG	Yes
HandEcompressed_Scan1.qptiff	512x512	None	No
HandEcompressed_Scan1.qptiff	512x512	JPEG	No

% du -csh *
272M	CMU-3.ndpi
 72K	CMU-3_2k
 72K	CMU-3_512
2.0G	CMU-3_j2k
190M	CMU-3_jpeg
 72K	CMU-3_nodual
190M	CMU-3_nodualjpeg
416M	HandEcompressed_Scan1.qptiff
4.6G	HandEcompressed_Scan1_2k
3.2G	HandEcompressed_Scan1_512
956M	HandEcompressed_Scan1_j2k
177M	HandEcompressed_Scan1_jpeg
3.2G	HandEcompressed_Scan1_nodual
177M	HandEcompressed_Scan1_nodualjpeg
 15G	total

all files generated with the dual personality option are valid TIFF as confirmed by tiffinfo
all files generated without the dual personality option fail tiffinfo with `Not a TIFF or MDI file, bad magic number 11825 (0x2e31)
the conversion of the CMU-3.npi file without compression shows an Exception in thread "main" loci.formats.FormatException: Cannot write more than 4GB of uncompressed pixel data. Specify a compression type instead. either with 512x512 or 2048x2048 tiles. The conversion produces 8.7K DICOM files which can are initialized by DicomReader but display corrupted pixel data
all other CMU-3 converted files (with compression) open without issue with DIcomReader and display valid pixel data
the converted HandEcompressed_Scan1 datasets with 512x512 or 2048x2048, no compression and the dual-personality option both fail initialization with an IllegalArgumentException

(java11) sbesson@sebastiensmbp2 DICOM % ~/Documents/GitHub/bioformats/tools/showinf -noflat -crop 10240,10240,2048,2048 HandEcompressed_Scan1_512/HandEcompressed_Scan1_512_0_0.dcm
Checking file format [DICOM]
Initializing reader
DicomReader initializing HandEcompressed_Scan1_512/HandEcompressed_Scan1_512_0_0.dcm
Verifying DICOM format
Reading tags
Exception in thread "main" java.lang.IllegalArgumentException: newPosition > limit: (25704 > 25703)
	at java.base/java.nio.Buffer.createPositionException(Buffer.java:318)
	at java.base/java.nio.Buffer.position(Buffer.java:293)
	at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1094)
	at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:262)
	at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:650)
	at loci.common.NIOFileHandle.seek(NIOFileHandle.java:330)
	at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:203)
	at loci.formats.in.DicomReader.initFile(DicomReader.java:676)
	at loci.formats.FormatReader.setId(FormatReader.java:1466)
	at loci.formats.ImageReader.setId(ImageReader.java:863)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:660)
	at loci.formats.tools.ImageInfo.testRead(ImageInfo.java:1043)
	at loci.formats.tools.ImageInfo.main(ImageInfo.java:1129)
(java11) sbesson@sebastiensmbp2 DICOM % ~/Documents/GitHub/bioformats/tools/showinf -noflat -crop 10240,10240,2048,2048 HandEcompressed_Scan1_2k/HandEcompressed_Scan1_2k_0_0.dcm                    
Checking file format [DICOM]
Initializing reader
DicomReader initializing HandEcompressed_Scan1_2k/HandEcompressed_Scan1_2k_0_0.dcm
Verifying DICOM format
Reading tags
Exception in thread "main" java.lang.IllegalArgumentException: newPosition > limit: (1944 > 1943)
	at java.base/java.nio.Buffer.createPositionException(Buffer.java:318)
	at java.base/java.nio.Buffer.position(Buffer.java:293)
	at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1094)
	at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:262)
	at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:650)
	at loci.common.NIOFileHandle.seek(NIOFileHandle.java:330)
	at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:203)
	at loci.formats.in.DicomReader.initFile(DicomReader.java:676)
	at loci.formats.FormatReader.setId(FormatReader.java:1466)
	at loci.formats.ImageReader.setId(ImageReader.java:863)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:660)
	at loci.formats.tools.ImageInfo.testRead(ImageInfo.java:1043)
	at loci.formats.tools.ImageInfo.main(ImageInfo.java:1129)

both datasets open without issue when selecting -format MinimalTiff
all other converted HandEcompressed_Scan1 open as expected

melissalinkert · 2023-07-27T21:54:16Z

Thanks for the reviews, @sbesson and @dgault. I could not reproduce the 2 errors in #3911 (review) with this PR's branch alone, but I can reproduce them with the most recent merge build's HEAD~ (801bddb).

I was very confused as to why that would be the case, but realized there is a 3 byte difference in length between files written with the merge build and files written with the branch. These 3 bytes are accounted for by the difference in the version number; 6.13.1-SNAPSHOT vs 7.0.0-SNAPSHOT. The string in TIFF metadata accounts for 1 byte, and the string in DICOM metadata accounts for 2 bytes (since strings are padded to a multiple of 2).

I believe the smaller version string combined with the implicit-length TRAILING_PADDING tag (which is only written for dual personality) and uncompressed pixel data creates an unexpected edge case that allows seeking beyond the end of the file. As far as I can tell, this is a reader issue only, and is fixed in 013b211 by assuming that the TRAILING_PADDING tag is at the end of the file.

melissalinkert · 2023-07-27T22:21:23Z

@fedorov / @dclunie: mentioned via email as well, but for completeness could you please let us know here or via private email what acknowledgement text you prefer to see in the release notes? #3742 (comment) is what we used in the past, but I'd assume that requires some changes.

sbesson

Retested the reading of all converted DICOM filesets created in #3911 (review) with the last commit.

Everything now works as expected with the only broken data being the CMU-3 files converted without compression and all other converted datasets initializing and reading binary data identical to the source file. Happy for this to get included in 7.0.0 in the current state.

Also specifically on #3911 (comment), retested the conversion of the NPDI file mentioned in the linked issue and the original image.sc thread. Confirmed that the PhotometricInterpretation is now set to YCbCr when using JPEG compression and RGB otherwise. Does that mean the associated issue should be closed by this PR?

(base) sbesson@sebastiensmbp2 Downloads % tiffinfo ndpi_he.ome.tiff | grep "Photometric Inter"
  Photometric Interpretation: RGB color
  Photometric Interpretation: RGB color
  Photometric Interpretation: RGB color
  Photometric Interpretation: RGB color
  Photometric Interpretation: min-is-black
  Photometric Interpretation: RGB color
  Photometric Interpretation: RGB color
  Photometric Interpretation: min-is-black
  Photometric Interpretation: RGB color
  Photometric Interpretation: min-is-black
(base) sbesson@sebastiensmbp2 Downloads % tiffinfo ndpi_he_jpeg.ome.tiff | grep "Photometric Inter"
  Photometric Interpretation: YCbCr
  Photometric Interpretation: YCbCr
  Photometric Interpretation: YCbCr
  Photometric Interpretation: YCbCr
  Photometric Interpretation: min-is-black
  Photometric Interpretation: YCbCr
  Photometric Interpretation: YCbCr
  Photometric Interpretation: min-is-black
  Photometric Interpretation: YCbCr
  Photometric Interpretation: min-is-black
(base) sbesson@sebastiensmbp2 Downloads % tiffinfo ndpi_he_j2k.ome.tiff | grep "Photometric Inter"
  Photometric Interpretation: RGB color
  Photometric Interpretation: RGB color
  Photometric Interpretation: RGB color
  Photometric Interpretation: RGB color
  Photometric Interpretation: min-is-black
  Photometric Interpretation: RGB color
  Photometric Interpretation: RGB color
  Photometric Interpretation: min-is-black
  Photometric Interpretation: RGB color
  Photometric Interpretation: min-is-black

melissalinkert · 2023-07-28T15:42:33Z

Also specifically on #3911 (comment), retested the conversion of the NPDI file mentioned in the #3856 and the original image.sc thread. Confirmed that the PhotometricInterpretation is now set to YCbCr when using JPEG compression and RGB otherwise. Does that mean the associated issue should be closed by this PR?

Agreed, I have updated the description so that #3856 should be closed automatically when this is merged.

melissalinkert added 5 commits November 18, 2022 12:58

Move the BigTIFF byte count constant to TiffConstants

ce1a97f

...so that other writers can use it too.

TiffSaver: add a writeIFD signature that calculates the next IFD's of…

3655d2b

…fset Useful for cases where all IFDs are written sequentially.

DicomWriter: add initial dual personality DICOM/TIFF support

4a2ad29

bfconvert: pass -bigtiff flag to DicomWriter

b020ee8

Fix length of trailing padding tag

ae3f1fa

Upload one set of artifacts from the GitHub Actions ant build

1572bae

melissalinkert added 2 commits November 21, 2022 10:54

Fix conditional to only upload one set of artifacts

0e65225

melissalinkert added 3 commits January 17, 2023 18:25

Merge branch 'develop' of github.com:openmicroscopy/bioformats into d…

79f22e0

…icom-dual-personality

Some improvements to uncompressed and JPEG-2000 tile writing

6d3dee9

JPEG still needs some work to be compliant with ImageMagick.

Fix the TIFF PhotometricInterpretation used with JPEG compression

63d75e0

See ome#3856

Merge branch 'develop' of github.com:openmicroscopy/bioformats into d…

5c18791

…icom-dual-personality

melissalinkert marked this pull request as ready for review March 21, 2023 14:51

Merge branch 'develop' of github.com:openmicroscopy/bioformats into d…

cdc63df

…icom-dual-personality

melissalinkert mentioned this pull request May 1, 2023

Allow reading and writing compressed tiles #3992

Merged

snoopycrimecop mentioned this pull request May 3, 2023

DICOM: fix raw RGB tile reading #3997

Merged

melissalinkert added this to the 6.14.0 milestone May 5, 2023

melissalinkert added the exclude label May 5, 2023

Merge branch 'develop' of github.com:openmicroscopy/bioformats into d…

e98ab3f

…icom-dual-personality

melissalinkert removed the exclude label Jun 16, 2023

Catch one more place where IFDs are incorrectly assumed to be non-null

0583e88

As noted in the comment, the IFDs will always be null if dual personality writing is turned off.

sbesson self-requested a review June 29, 2023 14:48

Fixes for reading files with more than 2GB of pixel data

d56bcdd

melissalinkert requested a review from sbesson June 30, 2023 16:44

sbesson approved these changes Jul 3, 2023

View reviewed changes

melissalinkert added the exclude label Jul 3, 2023

melissalinkert modified the milestones: 6.14.0, 7.0.0 Jul 3, 2023

melissalinkert mentioned this pull request Jul 3, 2023

Bio-Formats 6.14.0 version history ome/bio-formats-documentation#313

Merged

melissalinkert added 3 commits July 11, 2023 19:08

Fixes for handling run length encoded pixel data with undefined length

cad79aa

Fail fast when trying to write more than 4GB of uncompressed DICOM data

c891f10

Defend against empty JPEG/JPEG-2000 tiles

77c2ae2

melissalinkert removed the exclude label Jul 12, 2023

melissalinkert added 2 commits July 13, 2023 10:01

Move pixel count check to saveBytes

48ff25f

...since compression isn't set until after setId.

Fix off-by-one when checking if pixel data can be skipped

0f7d1c6

dgault approved these changes Jul 24, 2023

View reviewed changes

sbesson reviewed Jul 27, 2023

View reviewed changes

Add case for TRAILING_PADDING tag during parsing

013b211

sbesson approved these changes Jul 28, 2023

View reviewed changes

dgault mentioned this pull request Jul 28, 2023

Add dicom.dual_personality writer option ome/bio-formats-documentation#325

Merged

dgault merged commit c31c9ef into ome:develop Jul 31, 2023
17 checks passed

melissalinkert deleted the dicom-dual-personality branch September 6, 2024 19:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DICOM writer: initial support for dual personality DICOM/TIFF #3911

DICOM writer: initial support for dual personality DICOM/TIFF #3911

melissalinkert commented Nov 18, 2022 •

edited

Loading

joshmoore commented Nov 21, 2022

melissalinkert commented Nov 21, 2022 •

edited

Loading

joshmoore commented Nov 21, 2022

melissalinkert commented Nov 29, 2022 •

edited

Loading

melissalinkert commented Jan 26, 2023

melissalinkert commented Jun 16, 2023

dclunie commented Jun 18, 2023

melissalinkert commented Jun 20, 2023

melissalinkert commented Jun 30, 2023

dgault commented Jun 30, 2023

dgault commented Jun 30, 2023

sbesson left a comment

dgault commented Jul 3, 2023

melissalinkert commented Jul 3, 2023

melissalinkert commented Jul 12, 2023

melissalinkert commented Jul 13, 2023

dgault left a comment

sbesson left a comment

melissalinkert commented Jul 27, 2023

melissalinkert commented Jul 27, 2023

sbesson left a comment

melissalinkert commented Jul 28, 2023

DICOM writer: initial support for dual personality DICOM/TIFF #3911

DICOM writer: initial support for dual personality DICOM/TIFF #3911

Conversation

melissalinkert commented Nov 18, 2022 • edited Loading

joshmoore commented Nov 21, 2022

melissalinkert commented Nov 21, 2022 • edited Loading

joshmoore commented Nov 21, 2022

melissalinkert commented Nov 29, 2022 • edited Loading

melissalinkert commented Jan 26, 2023

melissalinkert commented Jun 16, 2023

dclunie commented Jun 18, 2023

melissalinkert commented Jun 20, 2023

melissalinkert commented Jun 30, 2023

dgault commented Jun 30, 2023

dgault commented Jun 30, 2023

sbesson left a comment

Choose a reason for hiding this comment

dgault commented Jul 3, 2023

melissalinkert commented Jul 3, 2023

melissalinkert commented Jul 12, 2023

melissalinkert commented Jul 13, 2023

dgault left a comment

Choose a reason for hiding this comment

sbesson left a comment

Choose a reason for hiding this comment

melissalinkert commented Jul 27, 2023

melissalinkert commented Jul 27, 2023

sbesson left a comment

Choose a reason for hiding this comment

melissalinkert commented Jul 28, 2023

melissalinkert commented Nov 18, 2022 •

edited

Loading

melissalinkert commented Nov 21, 2022 •

edited

Loading

melissalinkert commented Nov 29, 2022 •

edited

Loading