topic Re: Problems with starting Development Engironment in HPE Ezmeral Software platform

Problems with starting Development Engironment

rbukarev — Thu, 27 Jul 2023 10:53:43 GMT

I'm tryign to set up an HPE Ezmeral Data Fabric Development Environment as described here. Having some issues though:

1. When installing on Linux, I can't install mapr-client package, the error by RPM installer is:

sudo rpm -ivh mapr-client-7.3.0.0.20230425002320.GA-1.x86_64.rpm warning: mapr-client-7.3.0.0.20230425002320.GA-1.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 66b3f0d6: NOKEY error: Failed dependencies: lsof is needed by mapr-client-7.3.0.0.20230425002320.GA-1.x86_64 mapr-hadoop-util >= 2.7.6.0 is needed by mapr-client-7.3.0.0.20230425002320.GA-1.x86_64 mapr-librdkafka >= 0.11.3 is needed by mapr-client-7.3.0.0.20230425002320.GA-1.x86_64 syslinux is needed by mapr-client-7.3.0.0.20230425002320.GA-1.x86_64

What am I supposed to do about mapr-hadoop-util and mapr-librdkafka?

2. When installing on Mac, the docker container fails to start all services. This is what I see in maprcli-root-0.log:

Header: hostName: maprdemo.mapr.io, Time Zone: Coordinated Universal Time, processName: MapRCLI, processId: 2860, MapR Build Version: 7.3.0.0.20230425002320.GA 2023-07-26 01:00:13,741 INFO com.mapr.cliframework.driver.CLIMainDriver [main]: [node, cldbmaster] 2023-07-26 01:00:13,789 ERROR com.mapr.cliframework.base.CLIBaseClass setUserCredentials [main]: Problem with TicketKey file. Could not determine userID and GIDs of ticket, com.mapr.cliframework.base.CLIProcessingException: GetTicketAndKey returns error: 2 at com.mapr.cliframework.base.CLIBaseClass.setUserCredentials(CLIBaseClass.java:552) at com.mapr.cliframework.base.CLIBaseClass.<init>(CLIBaseClass.java:83) at com.mapr.cli.common.ListCommand.<init>(ListCommand.java:30) at com.mapr.cli.ServerCommands.<init>(ServerCommands.java:983) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at com.mapr.cliframework.base.CLICommandFactory.getCLI(CLICommandFactory.java:103) at com.mapr.cliframework.driver.CLIMainDriver.main(CLIMainDriver.java:122) 2023-07-26 01:00:14,244 ERROR com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils init [main]: Error 2 in loading /opt/mapr/conf/maprserverticket 2023-07-26 01:00:14,343 ERROR com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils getDataForParticularCLDB [main]: com.mapr.security.MaprSecurityException: You do not have a ticket to communicate with 127.0.0.1:7222. Retry after obtaining a new ticket using maprlogin at com.mapr.fs.jni.RpcNative.sendRequest(Native Method) at com.mapr.fs.Rpc.sendRequest(Rpc.java:107) at com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils.getDataForParticularCLDB(CLDBRpcCommonUtils.java:633) at com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils.sendRequest(CLDBRpcCommonUtils.java:526) at com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils.sendRequest(CLDBRpcCommonUtils.java:456) at com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils.sendRequest(CLDBRpcCommonUtils.java:448) at com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils.sendRequest(CLDBRpcCommonUtils.java:405) at com.mapr.cli.common.PluggableAlarmUtil.getAlarms(PluggableAlarmUtil.java:75) at com.mapr.cli.common.PluggableAlarmUtil.getAlarmsInternally(PluggableAlarmUtil.java:171) at com.mapr.cli.common.PluggableAlarmUtil.appendNodeMap(PluggableAlarmUtil.java:252) at com.mapr.cli.ServerCommands.executeRealCommand(ServerCommands.java:1003) at com.mapr.cliframework.base.CLIBaseClass.executeCommand(CLIBaseClass.java:255) at com.mapr.cliframework.driver.CLIMainDriver.main(CLIMainDriver.java:223)

Is there any step I'm missing?

Re: Problems with starting Development Engironment

ldarby — Thu, 27 Jul 2023 14:17:07 GMT

Hi @rbukarev ,

You can get these packages in the EEP repo (Ezmeral Ecosystem Packs, formerly MapR Ecosystem Packs) here:

https://package.mapr.hpe.com/releases/MEP/MEP-9.1.1/redhat/mapr-hadoop-util-3.3.4.200.202304060556-1.x86_64.rpm

https://package.mapr.hpe.com/releases/MEP/MEP-9.1.1/redhat/mapr-librdkafka-0.11.3.202201311452-1.x86_64.rpm

(assuming you're using Redhat (or derived), to get other OS packages go to .../MEP/MEP-9.1.1/)

It should also be possible to setup the repo as described here: https://docs.ezmeral.hpe.com/datafabric-customer-managed/73/AdvancedInstallation/UsingMapRInternetRepos.html then

yum install mapr-client

should work (and deal with any other dependencies). I'll get the docs updated about this.

For the 2nd issue, it looks like /opt/mapr/conf/maprserverticket didn't get generated, please check /opt/mapr/logs/configure.log in the container for any issues. On my system that worked, that log has these lines:

2023-07-27 09:46:13.764 maprdemo configure.sh(58) Install GenerateServerTicket:1697 Generating server ticket 2023-07-27 09:46:15.659 maprdemo configure.sh(58) Install ConfigureCLDBRole:1835 server ticket /opt/mapr/conf/maprserverticket is successfully generated

Regards,
Laurence Darby

Re: Problems with starting Development Engironment

rbukarev — Fri, 28 Jul 2023 02:36:53 GMT

Hi @ldarby thank you for looking into this!

1. On CentOS, I tried to follow the document on adding MapR Tech repository, but it seems that the document is missing the template for maprtech.repo file:

Create a text file called maprtech.repo in the /etc/yum.repos.d/ directory with the following content, replacing <version> with the version of data-fabric software that you want to install: (For the correct paths for all past releases, see the Data Fabric Repositories and Packages.)

2. As for the configure.log file, in there I have inly two lines:

2023-07-26 00:59:33.131 maprdemo configure.sh(153) Install check_valid_ip_list:2499 Using 7222 port for CLDB -Z 2023-07-26 00:59:33.310 maprdemo.mapr.io configure.sh(153) Install check_valid_ip_list:2499 No Zookeeper nodes list was provided. Exiting

Shouldn't the sript mapr_devsandbox_container_setup.sh have taken care of everything related to the container?

Thanks!

/RB

Re: Problems with starting Development Engironment

ldarby — Fri, 28 Jul 2023 11:43:18 GMT

Hi @rbukarev,

Apologies about the repo, there are changes planned soon and in advance of this the maprtech.repo contents were removed from the docs, it will be updated again when the repo changes happen. You can still just use rpm -i with the individual rpm downloads (and I've confirmed there aren't any further dependencies).

2) The -Z in 'CLDB -Z' is supposed to be an IP address, which means the start up script (/usr/bin/init-script) ran the configure.sh script with args '-C -Z' instead of '-C <ip> -Z', which means it couldn't find the IP address - maybe the networking didn't get initialised properly? Can you 'ip addr' inside the container with docker exec and show that please?

Regards,
Laurence Darby

Re: Problems with starting Development Engironment

rbukarev — Fri, 28 Jul 2023 12:39:49 GMT

Hi @ldarby , thank you for advice about Linux RPMs, I was able to install mapr-client on Centos7 and tried to install the docker container using script.. I got different error than on MacOS

But, first, here's what I see in the container on MacOS, when running 'ip adr':

root@maprdemo:/opt/mapr/logs# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000 link/tunnel6 :: brd :: 9: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default request send failed: Operation not supported link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff

The container in CentOS doesn't work, but apparently it went a bit further:

root@maprdemo:/opt/mapr/logs# jps 20777 FsShell 13273 QuorumPeerMain 20970 Jps 20572 CLDB 13662 WardenMain

Here's a few tailing records from /opt/mapr/logs/configure.log:

2023-07-28 12:21:12.863 maprdemo configure.sh(52) Install ConfigureEcoPackages:2186 Warning: Unable to run configure.sh for eco AWSMarketplace - not found - please implement 2023-07-28 12:21:12.875 maprdemo configure.sh(52) Install ConfigureDare:1787 Configuring dare 2023-07-28 12:21:12.886 maprdemo configure.sh(52) Install ConfigureDare:1790 Skipped configuration of dare, option -dare not found. 2023-07-28 12:21:16.649 maprdemo configure.sh(52) Install RunDiskSetup:547 Running disksetup: "/opt/mapr/server/disksetup -F /root/disk.list" 2023-07-28 12:21:22.249 maprdemo configure.sh(52) Install main:4415 DiskSetup failed ..

(not sure why it talks about AWSMarketplace, my CentOS VM is in GCP)

Thank you!

/RB

Re: Problems with starting Development Engironment

ldarby — Thu, 03 Aug 2023 17:38:57 GMT

Hi @rbukarev,

Apologies for the delay, did you get any further with this? On the Mac it has an error 'request send failed: Operation not supported' instead of the IP address, I'm not sure about this, the only thing I can find is this page with no answers: https://stackoverflow.com/questions/73576763/ip-netns-returns-operation-not-supported-in-docker-container-on-m1-mac.

On Centos, please check /opt/mapr/logs/disksetup.0.log in the container for any errors about formatting the disks. The script creates a 20GB sparse file at /opt/mapr/disks/docker.disk and uses that for a virtual disk.

The AWSMarketplace log message is odd but as far as I can tell it can be ignored.

Regards,
Laurence Darby

Re: Problems with starting Development Engironment

rbukarev — Fri, 04 Aug 2023 06:58:57 GMT

Hi @ldarby , let's pause the Mac-based container, maybe it's really M1-related issues.

CentOS-wise, here's the log in disksetup0.log

2023-08-04 06:34:45,763 12871 InitLogging:173 INFO ***** Starting disksetup: PID 12871 ***** 2023-08-04 06:34:45,764 12871 RunCmd:331 INFO uname -r: uname -r 2023-08-04 06:34:45,768 12871 <module>:1623 INFO argv: ['/opt/mapr/server/disksetup', '-F', '/root/disk.list'] 2023-08-04 06:34:45,768 12871 <module>:1624 INFO Options: Using -F force option 2023-08-04 06:34:45,768 12871 LoadDiskTab:686 INFO No disktab file: /opt/mapr/conf/disktab 2023-08-04 06:34:45,768 12871 SetCldbRole:1363 INFO Create cid = 1 for cldb role 2023-08-04 06:34:45,768 12871 LoadInputFile:709 INFO Start parsing input file: /root/disk.list 2023-08-04 06:34:45,768 12871 LoadInputFile:713 INFO /opt/mapr/disks/docker.disk 2023-08-04 06:34:45,768 12871 LoadInputFile:738 INFO Loaded disks: 2023-08-04 06:34:45,768 12871 LoadInputFile:740 INFO ([], 0, ['/opt/mapr/disks/docker.disk'], 1, '/opt/mapr/disks/docker.disk') 2023-08-04 06:34:45,769 12871 GetMfsUp:1370 INFO Lookup for : /opt/mapr/conf/mapr-clusters.conf 2023-08-04 06:34:45,769 12871 RunCmd:331 INFO mapr-mfs status: /opt/mapr/initscripts/mapr-mfs status 2023-08-04 06:34:45,968 12871 RunCmd:342 ERROR rc=3 2023-08-04 06:34:45,968 12871 RunCmd:343 ERROR * mfs is not running 2023-08-04 06:34:45,968 12871 RunCmd:346 INFO mapr-mfs status tried. err 3 2023-08-04 06:34:45,968 12871 RunCmd:331 INFO Fileserver start: /opt/mapr/initscripts/mapr-mfs start 2023-08-04 06:34:46,318 12871 RunCmd:331 INFO Disk list -v -n : /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660 disk list -v -n 2023-08-04 06:34:50,034 12871 LogCurrentInfo:1323 INFO start setup Trying to capture debug info, following errors can be ignored 2023-08-04 06:34:50,034 12871 RunCmd:331 INFO Disk list : /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660 disk list 2023-08-04 06:34:50,057 12871 LogCurrentInfo:1328 INFO ListDisks resp: status 0 count=0 2023-08-04 06:34:50,0542 :2907 MAPR_EXTERNAL: Empty string found 2023-08-04 06:34:50,057 12871 RunCmd:331 INFO DiskGroup list : /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660 dg list 2023-08-04 06:34:50,078 12871 LogCurrentInfo:1333 INFO No. of DGs (0) 2023-08-04 06:34:50,0758 :2907 MAPR_EXTERNAL: Empty string found 2023-08-04 06:34:50,079 12871 RunCmd:331 INFO sp list : /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660 sp list 2023-08-04 06:34:50,104 12871 LogCurrentInfo:1338 INFO ListSPs resp: status 0:0 2023-08-04 06:34:50,1016 :2907 MAPR_EXTERNAL: Empty string found 2023-08-04 06:34:50,1030 ERROR Global mrconfig.cc:790 No SP on this disk 2023-08-04 06:34:50,104 12871 LogCurrentInfo:1340 INFO Done capturing debug info 2023-08-04 06:34:50,104 12871 RunCmd:331 INFO Change mode: chmod g+rw /opt/mapr/disks/docker.disk: chmod g+rw /opt/mapr/disks/docker.disk 2023-08-04 06:34:50,107 12871 RunCmd:331 INFO Change group /opt/mapr/disks/docker.disk: chgrp mapr /opt/mapr/disks/docker.disk 2023-08-04 06:34:50,111 12871 InitDisk:455 INFO /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660 disk init -F /opt/mapr/disks/docker.disk 2023-08-04 06:34:50,111 12871 RunCmd:331 INFO Disk init /opt/mapr/disks/docker.disk: /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660 disk init -F /opt/mapr/disks/docker.disk 2023-08-04 06:34:50,138 12871 RunCmd:331 INFO Change mode: chmod g+rw /opt/mapr/disks/docker.disk: chmod g+rw /opt/mapr/disks/docker.disk 2023-08-04 06:34:50,141 12871 RunCmd:331 INFO Change group /opt/mapr/disks/docker.disk: chgrp mapr /opt/mapr/disks/docker.disk 2023-08-04 06:34:50,143 12871 RunCmd:331 INFO Disk load /opt/mapr/disks/docker.disk: /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660 disk load /opt/mapr/disks/docker.disk 2023-08-04 06:34:50,158 12871 RunCmd:331 INFO Disk list -v -n /opt/mapr/disks/docker.disk: /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660 disk list -v -n /opt/mapr/disks/docker.disk 2023-08-04 06:34:50,171 12871 UpdateDiskInfo:589 INFO guid 683CD029-FA89-0CD5-A0FF-010A9CCC6400 ListDisks resp: status 0 count=1 ListDisks /opt/mapr/disks/docker.disk guid 683CD029-FA89-0CD5-A0FF-010A9CCC6400 size 20480MB Numaid -1 DG 0: Single SingleDisk1 Online dgguid 683CD029-FA89-0CD5-A0FF-010A9CCC6400 2023-08-04 06:34:50,1690 :2907 MAPR_EXTERNAL: Empty string found 2023-08-04 06:34:50,172 12871 LogDiskInfoFile:1472 INFO Disk : docker.disk Vendor : unknown_vendor Model Number : unknown_model Serial Number : unknown_serial Firmware Revision : unknown_firmware Size : unknown_total 2023-08-04 06:34:50,172 12871 LogDiskInfoFile:1479 INFO Log to /opt/mapr/server/data/_opt_mapr_disks_docker.disk.info 2023-08-04 06:34:50,172 12871 RunCmd:331 INFO Change owner /opt/mapr/server/data/_opt_mapr_disks_docker.disk.info: chown mapr:mapr /opt/mapr/server/data/_opt_mapr_disks_docker.disk.info 2023-08-04 06:34:50,175 12871 LogDiskInfoFile:1484 INFO disk info logged in /opt/mapr/server/data/_opt_mapr_disks_docker.disk.info 2023-08-04 06:34:50,175 12871 SortDisks:866 INFO Sorting disks with size 2023-08-04 06:34:50,175 12871 RunCmd:331 INFO Disk remove /opt/mapr/disks/docker.disk: /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660 disk remove /opt/mapr/disks/docker.disk 2023-08-04 06:34:50,189 12871 GroupDisksAndCreateSPs:898 INFO Grouping disks create Storagepools 2023-08-04 06:34:50,189 12871 GroupDisksAndCreateSPs:915 INFO disk group sizes [1] 2023-08-04 06:34:50,190 12871 RunCmd:331 INFO SP NewSlot: /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660 sp getslot 2023-08-04 06:34:50,204 12871 LogExceptionHook:283 ERROR Generic exception handler: Traceback (most recent call last): File "/opt/mapr/server/disksetup", line 1636, in <module> RunDiskSetup(); File "/opt/mapr/server/disksetup", line 1510, in RunDiskSetup GroupDisksAndCreateSPs(force); File "/opt/mapr/server/disksetup", line 922, in GroupDisksAndCreateSPs ResetConfigCmdForInstance(diskList[0].numaid); File "/opt/mapr/server/disksetup", line 431, in ResetConfigCmdForInstance instancePort = int(output); ValueError: invalid literal for int() with base 10: '5660 2023-08-04 06:34:50,2012 :2907 MAPR_EXTERNAL: Empty string found ' 2023-08-04 06:34:50,204 12871 AbortWithError:326 ERROR Error 117, Structure needs cleaning. Traceback (most recent call last): File "/opt/mapr/server/disksetup", line 1636, in <module> RunDiskSetup(); File "/opt/mapr/server/disksetup", line 1510, in RunDiskSetup GroupDisksAndCreateSPs(force); File "/opt/mapr/server/disksetup", line 922, in GroupDisksAndCreateSPs ResetConfigCmdForInstance(diskList[0].numaid); File "/opt/mapr/server/disksetup", line 431, in ResetConfigCmdForInstance instancePort = int(output); ValueError: invalid literal for int() with base 10: '5660 2023-08-04 06:34:50,2012 :2907 MAPR_EXTERNAL: Empty string found ' [' File "/opt/mapr/server/disksetup", line 284, in LogExceptionHook\n AbortWithError(errno.EUCLEAN, "\\n" + text);\n', ' File "/opt/mapr/server/disksetup", line 324, in AbortWithError\n stack_trace = traceback.format_stack(frame)\n'] 2023-08-04 06:34:50,205 12871 LogCurrentInfo:1323 INFO Abort Trying to capture debug info, following errors can be ignored 2023-08-04 06:34:50,205 12871 RunCmd:331 INFO Disk list : /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660 disk list 2023-08-04 06:34:50,218 12871 LogCurrentInfo:1328 INFO ListDisks resp: status 0 count=0 2023-08-04 06:34:50,2161 :2907 MAPR_EXTERNAL: Empty string found 2023-08-04 06:34:50,218 12871 RunCmd:331 INFO DiskGroup list : /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660 dg list 2023-08-04 06:34:50,232 12871 LogCurrentInfo:1333 INFO No. of DGs (0) 2023-08-04 06:34:50,2296 :2907 MAPR_EXTERNAL: Empty string found 2023-08-04 06:34:50,232 12871 RunCmd:331 INFO sp list : /opt/mapr/server/mrconfig -h 127.0.0.1 -p 5660 sp list 2023-08-04 06:34:50,245 12871 LogCurrentInfo:1338 INFO ListSPs resp: status 0:0 2023-08-04 06:34:50,2427 :2907 MAPR_EXTERNAL: Empty string found 2023-08-04 06:34:50,2441 ERROR Global mrconfig.cc:790 No SP on this disk 2023-08-04 06:34:50,245 12871 LogCurrentInfo:1340 INFO Done capturing debug info 2023-08-04 06:34:50,246 12871 RestoreUdevRules:968 INFO Udev restoration not required. 2023-08-04 06:34:50,246 12871 RunCmd:331 INFO Fileserver stop: /opt/mapr/initscripts/mapr-mfs stop 2023-08-04 06:34:51,457 12871 ExitDiskSetup:294 ERROR Error 117, Structure needs cleaning. Traceback (most recent call last): File "/opt/mapr/server/disksetup", line 1636, in <module> RunDiskSetup(); File "/opt/mapr/server/disksetup", line 1510, in RunDiskSetup GroupDisksAndCreateSPs(force); File "/opt/mapr/server/disksetup", line 922, in GroupDisksAndCreateSPs ResetConfigCmdForInstance(diskList[0].numaid); File "/opt/mapr/server/disksetup", line 431, in ResetConfigCmdForInstance instancePort = int(output); ValueError: invalid literal for int() with base 10: '5660 2023-08-04 06:34:50,2012 :2907 MAPR_EXTERNAL: Empty string found ' [' File "/opt/mapr/server/disksetup", line 284, in LogExceptionHook\n AbortWithError(errno.EUCLEAN, "\\n" + text);\n', ' File "/opt/mapr/server/disksetup", line 324, in AbortWithError\n stack_trace = traceback.format_stack(frame)\n']

meanwhile I can see the file /opt/mapr/disks/docker.disk in the container. There's still 13GB free on the host machine's disk.

I tried to follow up the "MAPR_EXTERNAL: EMpty string found" error, the troubleshooting document recommends running the setup script with additional parameter -nwiterface enp4s0, but it didn't help (actually, the log above is after I tried this). Any other ideas?

Re: Problems with starting Development Engironment

rbukarev — Wed, 27 Sep 2023 06:50:07 GMT

@ldarby

So, I realised that in the option -nwiterface enp4s0, the enp4s0 was just an exmple. So I used eth0. And got the following error in the docker logs:

ERROR (22) - You do not have a ticket to communicate with 172.17.0.2:7222. Retry after obtaining a new ticket using maprlogin after cldb Wed Sep 27 05:27:09 UTC 2023

But apparently I can access https://localhost:8443 -- does it mean it actually works?

Re: Problems with starting Development Engironment

ldarby — Wed, 27 Sep 2023 09:34:57 GMT

Hi @rbukarev, apologies, yes those docs about enp4s0 aren't clear, I'll get it updated. That error message usually happens when you run a maprcli command without a ticket, and is solved by generating a ticket with this command:

maprlogin password

However it's not clear why that error is in the docker logs, possibly the system generated tickets (in /opt/mapr/conf/mapruserticket and /opt/mapr/conf/maprserverticket) didn't get generated properly?