I have very recently learned about zram-swap. Fedora-29-arm is using it. It supposedly makes a considerable difference even over HD swap.
Since many (potential) users of Centos-arm do not have a ready HD, perhaps we can switch the image from a physical partition to zram? Or at least instructions to change to this on the wiki.
Perhaps target the 7.6 image? (is there any information as to when 7.6 may be available?)
thanks
Too many questions together... ;-)
El 7/10/18 a las 08:58, Robert Moskowitz escribió:
I have very recently learned about zram-swap. Fedora-29-arm is using it. It supposedly makes a considerable difference even over HD swap.
Do you have a link for me to read about that? I'd like to see some numbers on performance, specially on the CPU, because small ARM devices are not exactly fast to compress.
Since many (potential) users of Centos-arm do not have a ready HD, perhaps we can switch the image from a physical partition to zram? Or at least instructions to change to this on the wiki.
Switch might be difficult, but maybe generate a "tech preview" alternative build. This is the link to how we create the images, maybe you could send me some changes to test https://github.com/CentOS/sig-core-AltArch/tree/master/image_build
Perhaps target the 7.6 image? (is there any information as to when 7.6 may be available?)
The usual response we get is, you'll know when it is there...
thanks
Pablo.
On 10/7/18 11:57 AM, Pablo Sebastián Greco wrote:
Too many questions together... ;-)
El 7/10/18 a las 08:58, Robert Moskowitz escribió:
I have very recently learned about zram-swap. Fedora-29-arm is using it. It supposedly makes a considerable difference even over HD swap.
Do you have a link for me to read about that? I'd like to see some numbers on performance, specially on the CPU, because small ARM devices are not exactly fast to compress.
You could ask on the Fedora-arm list where Peter Robinson (I believe he is part of the Fedora development with heavy presence in the arm work) recently explained, kind of, how F29-arm is using it.
Since many (potential) users of Centos-arm do not have a ready HD, perhaps we can switch the image from a physical partition to zram? Or at least instructions to change to this on the wiki.
Switch might be difficult, but maybe generate a "tech preview" alternative build. This is the link to how we create the images, maybe you could send me some changes to test https://github.com/CentOS/sig-core-AltArch/tree/master/image_build
Minimally in the wiki, how to switch from a partition swap to zram. I am working on learning more about it...
I have been working with Mark Verlinde, well actually he did the heavy lifting, to port zram to Centos...
https://copr.fedorainfracloud.org/coprs/markvnl/zram-swap/
A few of us are actively using it.
Give it a try yourselves!
As it is noarch, it should work on all Centos7 arch.
On 10/7/18 1:06 PM, Robert Moskowitz wrote:
On 10/7/18 11:57 AM, Pablo Sebastián Greco wrote:
Too many questions together... ;-)
El 7/10/18 a las 08:58, Robert Moskowitz escribió:
I have very recently learned about zram-swap. Fedora-29-arm is using it. It supposedly makes a considerable difference even over HD swap.
Do you have a link for me to read about that? I'd like to see some numbers on performance, specially on the CPU, because small ARM devices are not exactly fast to compress.
You could ask on the Fedora-arm list where Peter Robinson (I believe he is part of the Fedora development with heavy presence in the arm work) recently explained, kind of, how F29-arm is using it.
Since many (potential) users of Centos-arm do not have a ready HD, perhaps we can switch the image from a physical partition to zram? Or at least instructions to change to this on the wiki.
Switch might be difficult, but maybe generate a "tech preview" alternative build. This is the link to how we create the images, maybe you could send me some changes to test https://github.com/CentOS/sig-core-AltArch/tree/master/image_build
Minimally in the wiki, how to switch from a partition swap to zram. I am working on learning more about it...
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
You do realize that some of us have been using swap-on-zram for the best part of a decade, right? It never required a systemd service, you just need an entry in /etc/modprobe.d (optional), an entry in fastab (optionl) and a script in /etc/sysconfig/modules (you can fold the other two I mentioned in here so it's in a single place). At the beginning zram only used LZO compression, and was single-threaded, so we created one zram block device per CPU core, did mkswap on them and did swapon with equap priority on all of those swap devices. I had this on on Fedora and later RSEL6 on my SheevaPlug Toshiba AC100 as far back as 2009, and I can't have been the only one.
In fact, this related ticket dates back to 2007, and I only got in on it in 2009: https://bugzilla.redhat.com/show_bug.cgi?id=223722
Apparently that was a "can't fix" - worked for me since 2009 with the solution I attached to that ticket.
On Fri, Oct 12, 2018 at 4:47 PM Robert Moskowitz rgm@htt-consult.com wrote:
I have been working with Mark Verlinde, well actually he did the heavy lifting, to port zram to Centos...
https://copr.fedorainfracloud.org/coprs/markvnl/zram-swap/
A few of us are actively using it.
Give it a try yourselves!
As it is noarch, it should work on all Centos7 arch.
On 10/7/18 1:06 PM, Robert Moskowitz wrote:
On 10/7/18 11:57 AM, Pablo Sebastián Greco wrote:
Too many questions together... ;-)
El 7/10/18 a las 08:58, Robert Moskowitz escribió:
I have very recently learned about zram-swap. Fedora-29-arm is using it. It supposedly makes a considerable difference even over HD swap.
Do you have a link for me to read about that? I'd like to see some numbers on performance, specially on the CPU, because small ARM devices are not exactly fast to compress.
You could ask on the Fedora-arm list where Peter Robinson (I believe he is part of the Fedora development with heavy presence in the arm work) recently explained, kind of, how F29-arm is using it.
Since many (potential) users of Centos-arm do not have a ready HD, perhaps we can switch the image from a physical partition to zram? Or at least instructions to change to this on the wiki.
Switch might be difficult, but maybe generate a "tech preview" alternative build. This is the link to how we create the images, maybe you could send me some changes to test https://github.com/CentOS/sig-core-AltArch/tree/master/image_build
Minimally in the wiki, how to switch from a partition swap to zram. I am working on learning more about it...
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
On 10/7/18 11:57 AM, Pablo Sebastián Greco wrote:
Too many questions together... ;-)
El 7/10/18 a las 08:58, Robert Moskowitz escribió:
I have very recently learned about zram-swap. Fedora-29-arm is using it. It supposedly makes a considerable difference even over HD swap.
Do you have a link for me to read about that? I'd like to see some numbers on performance, specially on the CPU, because small ARM devices are not exactly fast to compress.
I did a quick check of my F29-arm system's use of zram. It is running zram 0.2-1. A quick look at usage with an Xfce DE accessed via vnc that has Firefox running:
# swapon -s Filename Type Size Used Priority /dev/zram0 partition 487304 10752 -2
# zramctl NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT /dev/zram0 lz4 475.9M 11.2M 4.3M 4.9M 2 [SWAP]
I would suspect the performance of lz4 over SD I/O is superior, plus less wear and tear of memory over SD. Here are a couple files and the systemctl status:
# cat /etc/zram.conf # The factor is the percentage of total system RAM to allocate to the ZRAM block device(s). FACTOR=2
# cat /etc/systemd/system/swap.target.wants/zram-swap.service [Unit] Description=Enable compressed swap in memory using zram DefaultDependencies=no Before=swap.target
[Service] Type=oneshot RemainAfterExit=yes TimeoutStartSec=30sec ExecStart=/usr/sbin/zramstart ExecStop=/usr/sbin/zramstop
[Install] WantedBy=swap.target
# systemctl -l --no-pager status zram-swap \u25cf zram-swap.service - Enable compressed swap in memory using zram Loaded: loaded (/usr/lib/systemd/system/zram-swap.service; enabled; vendor preset: disabled) Active: active (exited) since Fri 2018-06-22 11:12:12 EDT; 3 months 16 days ago Process: 437 ExecStart=/usr/sbin/zramstart (code=exited, status=0/SUCCESS) Main PID: 437 (code=exited, status=0/SUCCESS)
Jun 22 11:12:12 localhost zramstart[437]: Setting up swapspace version 1, size = 475.9 MiB (498999296 bytes) Jun 22 11:12:12 localhost zramstart[437]: no label, UUID=ba3bf62f-7bb0-4295-ad8c-2d26b000c3f8 Jun 22 11:12:12 localhost zramstart[437]: Activated ZRAM swap device of 499 MB Jun 22 11:12:12 localhost systemd[1]: Started Enable compressed swap in memory using zram.
On Oct 7, 2018, at 07:58, Robert Moskowitz rgm@htt-consult.com wrote:
I have very recently learned about zram-swap. Fedora-29-arm is using it. It supposedly makes a considerable difference even over HD swap.
Since many (potential) users of Centos-arm do not have a ready HD, perhaps we can switch the image from a physical partition to zram? Or at least instructions to change to this on the wiki.
Not to be utterly naive, but why is swap required on such systems at all? To take core RAM and repurpose it as swap would seem to vitiate the entire point of having a virtual memory subsystem in the first place.
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | Perfection is reached, not when there is no longer anything to add, | | but when there is no longer anything to take away. | | -- Antoine de Saint-Exupery | |----------------------------------------------------------------------|
Because swapping to SD card is slow, swapping to zram is fast, and zram only takes about 1/2 of the RAM that it provides. It's basically RAM compression. It works and makes a positive difference, even without exceedingly slow storage.
On Fri, Oct 12, 2018 at 5:16 PM Fred Gleason fredg@paravelsystems.com wrote:
On Oct 7, 2018, at 07:58, Robert Moskowitz rgm@htt-consult.com wrote:
I have very recently learned about zram-swap. Fedora-29-arm is using it. It supposedly makes a considerable difference even over HD swap.
Since many (potential) users of Centos-arm do not have a ready HD, perhaps we can switch the image from a physical partition to zram? Or at least instructions to change to this on the wiki.
Not to be utterly naive, but why is swap required on such systems at all? To take core RAM and repurpose it as swap would seem to vitiate the entire point of having a virtual memory subsystem in the first place.
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | Perfection is reached, not when there is no longer anything to add, | | but when there is no longer anything to take away. | | -- Antoine de Saint-Exupery | |----------------------------------------------------------------------| _______________________________________________ Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
It is now built into Fedora29-arm. That is where Mark got the source and the systemd support. Yes, earlier many just used it. Now Fedora-arm is based on it.
On the Fedora-arm list, Peter Robinson has said he is seeing 4-5 fold compression. That means that if you use a factor of 2 and have a 1GB memory SOC, that is effectively 2.5GB ram. Or use a factor of 4 for effectively 1.75GB and less 'swapping'. So you can use your 1GB duo core system more effectively, even if you currently have swap on a HD or SSD sata or USB attached.
On 10/12/18 12:25 PM, Gordan Bobic wrote:
Because swapping to SD card is slow, swapping to zram is fast, and zram only takes about 1/2 of the RAM that it provides. It's basically RAM compression. It works and makes a positive difference, even without exceedingly slow storage.
On Fri, Oct 12, 2018 at 5:16 PM Fred Gleason <fredg@paravelsystems.com mailto:fredg@paravelsystems.com> wrote:
On Oct 7, 2018, at 07:58, Robert Moskowitz <rgm@htt-consult.com <mailto:rgm@htt-consult.com>> wrote:I have very recently learned about zram-swap. Fedora-29-arm is using it. It supposedly makes a considerable difference even over HD swap. Since many (potential) users of Centos-arm do not have a ready HD, perhaps we can switch the image from a physical partition to zram? Or at least instructions to change to this on the wiki.Not to be utterly naive, but why is swap required on such systems at all? To take core RAM and repurpose it as swap would seem to vitiate the entire point of having a virtual memory subsystem in the first place. Cheers! |----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | Perfection is reached, not when there is no longer anything to add, | | but when there is no longer anything to take away. | | -- Antoine de Saint-Exupery | |----------------------------------------------------------------------| _______________________________________________ Arm-dev mailing list Arm-dev@centos.org <mailto:Arm-dev@centos.org> https://lists.centos.org/mailman/listinfo/arm-dev
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
Depends on his use case. The only way he is seeing 4-5:1 CR is if there are a lot of 0-filled pages being swapped out (those just get thrown away completely). Firefox is known to request tons of RAM this way, of which it regularly ends up not using most. On a more sane load, you will typically achieve about 2:1.
Realistically, if you have 1GB of RAM and give 512MB to zram, that gives you about 512MB of uncompressed RAM and 1GB of somewhat slower compressed RAM that is still way faster than swapping.
On Fri, Oct 12, 2018 at 5:46 PM Robert Moskowitz rgm@htt-consult.com wrote:
It is now built into Fedora29-arm. That is where Mark got the source and the systemd support. Yes, earlier many just used it. Now Fedora-arm is based on it.
On the Fedora-arm list, Peter Robinson has said he is seeing 4-5 fold compression. That means that if you use a factor of 2 and have a 1GB memory SOC, that is effectively 2.5GB ram. Or use a factor of 4 for effectively 1.75GB and less 'swapping'. So you can use your 1GB duo core system more effectively, even if you currently have swap on a HD or SSD sata or USB attached.
On 10/12/18 12:25 PM, Gordan Bobic wrote:
Because swapping to SD card is slow, swapping to zram is fast, and zram only takes about 1/2 of the RAM that it provides. It's basically RAM compression. It works and makes a positive difference, even without exceedingly slow storage.
On Fri, Oct 12, 2018 at 5:16 PM Fred Gleason fredg@paravelsystems.com wrote:
On Oct 7, 2018, at 07:58, Robert Moskowitz rgm@htt-consult.com wrote:
I have very recently learned about zram-swap. Fedora-29-arm is using it. It supposedly makes a considerable difference even over HD swap.
Since many (potential) users of Centos-arm do not have a ready HD, perhaps we can switch the image from a physical partition to zram? Or at least instructions to change to this on the wiki.
Not to be utterly naive, but why is swap required on such systems at all? To take core RAM and repurpose it as swap would seem to vitiate the entire point of having a virtual memory subsystem in the first place.
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | Perfection is reached, not when there is no longer anything to add, | | but when there is no longer anything to take away. | | -- Antoine de Saint-Exupery | |----------------------------------------------------------------------| _______________________________________________ Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
Arm-dev mailing listArm-dev@centos.orghttps://lists.centos.org/mailman/listinfo/arm-dev
On Oct 12, 2018, at 12:25, Gordan Bobic gordan@redsleeve.org wrote:
Because swapping to SD card is slow, swapping to zram is fast, and zram only takes about 1/2 of the RAM that it provides. It's basically RAM compression. It works and makes a positive difference, even without exceedingly slow storage.
Right, but that begs a larger question: why swap *at all*? The whole point of virtual memory in the first place was as a performance hack to allow the processor to see a larger memory space than can be accommodated by core RAM alone. For certain workloads, in situations where one has a mass storage device upon which to place the swapped memory pages, this makes sense; but when the ‘storage device’ is the very core RAM that one is attempting to ‘extend’, it becomes a completely self-defeating exercise; complexity for the sake of complexity.
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | An expert is a person who avoids the small errors while he sweeps | | on to the grand fallacy. | | -- Benjamin Stolberg | |----------------------------------------------------------------------|
Because bloatware and OOM killer?
On Fri, Oct 12, 2018 at 6:03 PM Fred Gleason fredg@paravelsystems.com wrote:
On Oct 12, 2018, at 12:25, Gordan Bobic gordan@redsleeve.org wrote:
Because swapping to SD card is slow, swapping to zram is fast, and zram only takes about 1/2 of the RAM that it provides. It's basically RAM compression. It works and makes a positive difference, even without exceedingly slow storage.
Right, but that begs a larger question: why swap *at all*? The whole point of virtual memory in the first place was as a performance hack to allow the processor to see a larger memory space than can be accommodated by core RAM alone. For certain workloads, in situations where one has a mass storage device upon which to place the swapped memory pages, this makes sense; but when the ‘storage device’ is the very core RAM that one is attempting to ‘extend’, it becomes a completely self-defeating exercise; complexity for the sake of complexity.
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | An expert is a person who avoids the small errors while he sweeps | | on to the grand fallacy. | | -- Benjamin Stolberg | |----------------------------------------------------------------------| _______________________________________________ Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
On Oct 12, 2018, at 13:04, Gordan Bobic gordan@redsleeve.org wrote:
Because bloatware and OOM killer?
Can you elaborate on this please? FWIW, I have been configuring systems with no external mass storage to operate with no swap whatever for several years now, with no apparent ill effects. Why would this be poor practice?
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | A room without books is like a body without a soul. | | -- Cicero | |----------------------------------------------------------------------|
It depends on what you are running on that system. If you have 1GB of RAM and want to run a desktop environment web browser, that'll go pretty poorly without swap.
On Fri, Oct 12, 2018 at 6:10 PM Fred Gleason fredg@paravelsystems.com wrote:
On Oct 12, 2018, at 13:04, Gordan Bobic gordan@redsleeve.org wrote:
Because bloatware and OOM killer?
Can you elaborate on this please? FWIW, I have been configuring systems with no external mass storage to operate with no swap whatever for several years now, with no apparent ill effects. Why would this be poor practice?
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | A room without books is like a body without a soul. | | -- Cicero | |----------------------------------------------------------------------|
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
On Oct 12, 2018, at 13:15, Gordan Bobic gordan@redsleeve.org wrote:
It depends on what you are running on that system. If you have 1GB of RAM and want to run a desktop environment web browser, that'll go pretty poorly without swap.
So how does putting swap on RAM help that situation? You’re not increasing the overall memory space by doing so, but merely partitioning it into two different buckets (while also adding the complexity and performance overhead of accessing the part that is now ‘swap’).
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | A room without books is like a body without a soul. | | -- Cicero | |----------------------------------------------------------------------|
Because the zram bucket is compressed and thus requires typically half the amount of real RAM. So if you have 1GB of RAM and set up a 512MB zram for swap, those 512MB when completely used up will typically only consume 256MB of RAM. So you will end up with 768MB of actual RAM, + 512MB of swap = 1.25GB of RAM instead of 1GB. Tweak ratios to suit your workload, but that's the basic gist of it.
On Fri, Oct 12, 2018 at 6:24 PM Fred Gleason fredg@paravelsystems.com wrote:
On Oct 12, 2018, at 13:15, Gordan Bobic gordan@redsleeve.org wrote:
It depends on what you are running on that system. If you have 1GB of RAM and want to run a desktop environment web browser, that'll go pretty poorly without swap.
So how does putting swap on RAM help that situation? You’re not increasing the overall memory space by doing so, but merely partitioning it into two different buckets (while also adding the complexity and performance overhead of accessing the part that is now ‘swap’).
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | A room without books is like a body without a soul. | | -- Cicero | |----------------------------------------------------------------------|
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
On Oct 12, 2018, at 13:28, Gordan Bobic gordan@redsleeve.org wrote:
Because the zram bucket is compressed and thus requires typically half the amount of real RAM. So if you have 1GB of RAM and set up a 512MB zram for swap, those 512MB when completely used up will typically only consume 256MB of RAM. So you will end up with 768MB of actual RAM, + 512MB of swap = 1.25GB of RAM instead of 1GB. Tweak ratios to suit your workload, but that's the basic gist of it.
Ah! Ok. That makes sense. Thank you.
One more question: is there any benefit to configuring a system to use swap *beyond* that of increasing the the global memory space —e.g. more efficient memory management or better operati
On Oct 12, 2018, at 13:28, Gordan Bobic <gordan@redsleeve.org mailto:gordan@redsleeve.org> wrote:
Because the zram bucket is compressed and thus requires typically half the amount of real RAM. So if you have 1GB of RAM and set up a 512MB zram for swap, those 512MB when completely used up will typically only consume 256MB of RAM. So you will end up with 768MB of actual RAM, + 512MB of swap = 1.25GB of RAM instead of 1GB. Tweak ratios to suit your workload, but that's the basic gist of it.
Ah! Ok. That makes sense. Thank you.
One more question: is there any benefit to configuring a system to use swap *beyond* that of increasing the the global memory space —e.g. more efficient memory management or better operation of the OOM killer?
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | A room without books is like a body without a soul. | | -- Cicero | |----------------------------------------------------------------------|
On 10/12/18 1:24 PM, Fred Gleason wrote:
On Oct 12, 2018, at 13:15, Gordan Bobic <gordan@redsleeve.org mailto:gordan@redsleeve.org> wrote:
It depends on what you are running on that system. If you have 1GB of RAM and want to run a desktop environment web browser, that'll go pretty poorly without swap.
So how does putting swap on RAM help that situation? You’re not increasing the overall memory space by doing so, but merely partitioning it into two different buckets (while also adding the complexity and performance overhead of accessing the part that is now ‘swap’).
See Gordon's earlier comment on Firefox's bad memory allocation. Zram is a big help there; I see this on my F29 system.
# free total used free shared buff/cache available Mem: 1022148 655520 107204 22284 259424 327636 Swap: 487304 253952 233352
# zramctl NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT /dev/zram0 lz4 475.9M 246.4M 109.1M 114.2M 2 [SWAP]
My mail server can be all over the place on memory and swap usage (it uses real swap file on sata) depending on what all the scanning software is doing and who is using roundcubemail.
So YMMV, and this should not hurt if your system does not need to swap.
# free total used free shared buff/cache available Mem: 2060396 189332 1189972 26432 681092 1766392 Swap: 982420 0 982420
# zramctl NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT /dev/zram0 lz4 959.4M 4K 64B 4K 2 <swap>
I may switch this setup to a Cubeboard2 (1GB mem) from the Cubietruck. I have a number of C2 and only a couple CT, so if it does not need the memory...
Cheers!
|----------------------------------------------------------------------| | Frederick F. Gleason, Jr. | Chief Developer | | | Paravel Systems | |----------------------------------------------------------------------| | A room without books is like a body without a soul. | | -- Cicero | |----------------------------------------------------------------------|
Arm-dev mailing list Arm-dev@centos.org https://lists.centos.org/mailman/listinfo/arm-dev
-
Fred Gleason -
Gordan Bobic -
Pablo Sebastián Greco -
Robert Moskowitz