Forum Link: Original Post

Town Hall with Ian, Daniel, Cyn and Jeska Linden
Topic: Estates

Daniel Linden: Buenas dias!
Daniel Linden: You've arrived early -- wanted to get a good seat?
Daniel Linden: That's the first test....

Cyn Linden: Sure
Cyn Linden: Hi Jes
Cyn Linden: Nice devil
Cyn Linden: Yep
Cyn Linden: Hi Alliez

Jeska Linden: Hey everyone

Daniel Linden: Thanks!

Jeska Linden: We'll start in just a few more minutes :)
Jeska Linden: Hello everyone! Welcome to the Technical Town Hall with Ian Linden, director of operations. Thanks for coming out and Happy Halloween! :)

Jeska Linden: First some guidelines for the discussion. We will be focusing this Town Hall on questions about technical issues, please don't submit off-topic questions, as they will be skipped over.

Jeska Linden: Ian is going to open with a brief introduction and then we will go into an open QA. Questions can be sent to me (Jeska Linden) in the form of an IM.

Ian Linden: I am?

Jeska Linden: Also, for those who have joined us here, I'd like to ask you to please hold down the chat and offers of friendship until after the meeting as it can be distracting.
Jeska Linden: Ian, take it away.

Ian Linden: uh.. all you concerns will be fixed in, uh, 1.8, I, uh, promise
Ian Linden: ;)
Ian Linden: actaully, since I know there are alot of performance questions
Ian Linden: I think everyone should read the docs we've posted
Ian Linden: link
Ian Linden:
Ian Linden: not very in-depth as yet but they'll grow
Ian Linden: and it's a good baseline
Ian Linden: err, that's all I've got, I'd rather just spend the time on questions

Jeska Linden: Excellent, send them on over to me in IM :)
Jeska Linden: Nexus Nash: What's the status on estate tools v2 ? (more stats (scripts, objects etc), more control to deed, etc)

Ian Linden: forthcoming, but not immediately
Ian Linden: I can't give a date, unfortunately
Ian Linden: I think really these features will come in several small bits
Ian Linden: instead of one big overhaul
Ian Linden: we're focusing right now on fixing the existing bugs
Ian Linden: and lowering the crash rate
Ian Linden: after that I think we'll be able to start improving those tools
Ian Linden: it's clear that they are key in the long term
Ian Linden: sorry if that sounds like a cop-out, the schedule just isn't firm
Ian Linden: next q?

Jeska Linden: Surreal Farber: Since a private sim represents substantial investment, when will be able to register with LL things such as survivorship and/or designated representative.

Ian Linden: good question, I have no idea

Daniel Linden: I'll take it!

Ian Linden: daniel?

Daniel Linden: We're working on plans as we speak to give you all a MUCH higher level of control...
Daniel Linden: ...administratively, over your estates.
Daniel Linden: Cyn and I will be making the rounds to personally speak with you about these plans...
Daniel Linden: ...and make sure we moving in the direction you want.

Ian Linden: another "coming soon" I guess

Daniel Linden: Once we've finalized the details...we'll move pretty quickly.
Daniel Linden: Stay tuned!

Jeska Linden: Cutter Rubio: What provisions, if any, is LL making for keeping the private sims up to date in terms of hardware performance?

Ian Linden: that I can answer
Ian Linden: right now we replace old machines on a rough schedule
Ian Linden: the basic plan is three years although we've been shorter than that so far
Ian Linden: ultimately we won't keep class 2 hardware in service forever
Ian Linden: if you're on a class that gets upgraded/replaced
Ian Linden: then you'll be moved up the latest stuff
Ian Linden: this makes the most sense for us from a deployment perspective

Bub Linden: Script run-time error
Bub Linden: Too Many Listens

Ian Linden: I can't give a timeline for the death of class 2, but hopefully it will be sooner than the 3-year anniversary for those machines
Ian Linden: we've considered offering a way to pay to move up ahead of schedule
Ian Linden: but it doesn't seem like that would make alot of sense
Ian Linden: given the high cost and relatively modest performance gains
Ian Linden: but feel free to disagree with that ;)
Ian Linden: next?

Jeska Linden: David Valentino: With the new and more accurate script count/readings, is there a target for max number of scripts we should be looking at for optimal sim performance? Say in a 75% prim usage sim?

Ian Linden: hmm
Ian Linden: unfortunately that depends more on what the scripts are doing than how many there are
Ian Linden: clearly whenever we get above 2500 or so things start to slow down
Ian Linden: but nominally when you run a ton of scripts all you'll do is slow down the other scripts
Ian Linden: not the sim
Ian Linden: that's the goal anyway
Ian Linden: it's just the number for the sim
Ian Linden: the old number, it turns out, failed to count scripts on link children
Ian Linden: oops
Ian Linden: so they tell me
Ian Linden: so, clearly we need better accounting/scheduling
Ian Linden: as mentioned in the first question
Ian Linden: next?

Jeska Linden: Nexus Nash: Any reason why TD gets hammered post 1.7 whenever there is the slightless load on the server?

Ian Linden: yeah I'd like to know too
Ian Linden: a
Ian Linden: actually, we're not running havok in lockstep w/ the rest of the sim
Ian Linden: nominally this is more efficient
Ian Linden: but if the sim drops below 45 fps then havok suffers as well
Ian Linden: and vice versa
Ian Linden: I think we're going to find there are efficiency bugs in the current code
Ian Linden: yeah, it's not as good as it should be
Ian Linden: I would say, give us some time to patch this release
Ian Linden: paulie: I'd like to put that Q in the queue, jeska ok?

Jeska Linden: Sure
Jeska Linden: Please send questions in IM form to me :)

Ian Linden: anway, we've done some grid-wide sim stats analysis
Ian Linden: and I think we're zeroing in on some problems
Ian Linden: I'd have to go offline to get more detail
Ian Linden: next q?

Jeska Linden: Mike Getting: Do you allocate faster servers to islands with a higher average load, or do all islands get whatever's current when they're first created?

Ian Linden: no, they get what they got when they were created
Ian Linden: this seems to be the most fair
Ian Linden: esp. since we can't move them between servers while they're running
Ian Linden: yet
Ian Linden: next q?

Jeska Linden: Anshe Chung: In Dreamland we have more than 300 people who own land deeds. People buy and sell those every day, but the process is manual due to lack of tools....
Jeska Linden: When will you recognize this existing market and provide tools that allow us to automate the trading ofdeeds in zoned sims?

Ian Linden: daniel, maybe you can comment on that?

Daniel Linden: Yes!
Daniel Linden: Anshe, this issue is in our plans for THIS quarter.

Jeska Linden: When will you recognize this existing market and provide tools that allow us to automate the trading ofdeeds in zoned sims?

Jeska Linden: opps

Ian Linden: I think this sort of thing is a clear goal for us

Daniel Linden: We plan to start creating and rollilng out these tools (and more) as they're complete.

Ian Linden: the mainland can't be the only estate where land deeds are automated
Ian Linden: again, look for these in a series of small steps
Ian Linden: I think the "one giant release" model has been shown to have some problems ;)

Jeska Linden: Mike Getting: How do we find out what class we're in?

Ian Linden: yeah, that'd be handy, eh
Ian Linden: tell you what, after this is over, I'll post that to the sim stats wiki page

Daniel Linden: We're going to solidify these type of arrangements and put the tools into Second Life -- it'll be better for owners, renters, vistors....this is a big goal for us NOW.

Ian Linden: right now you have to look at the hostname of the server in the About SL window
Ian Linden: I suppose it would be a good idea to report it directly
Ian Linden: next?

Jeska Linden: paulie Femto: I know LL claims no performance degradation between sims on the same server, but can we find out what estates we are stacked with so that we could arrange large events so as not to conflict with stacked neihgbors events?

Ian Linden: interesting
Ian Linden: you can find out, again, by matching the hostname in About SL
Ian Linden: that tells you, at least, if two are on the same machine
Ian Linden: although it's a bit of a pain
Ian Linden: yeah, but right now that's pretty much all you've got, unfortunately
Ian Linden: the number of sims stacked on a machine is generally fixed for each class
Ian Linden: at least
Ian Linden: anyway, sorry I can't offer a better method
Ian Linden: I will say, however, that generaly speaking you shouldn't find persistent issues w/ stacked sims
Ian Linden: keeping in mind that ALL class 3s are stacked and everyone knows they're a bit faster than class 2s
Ian Linden: which are NEVER stacked
Ian Linden: however, we do run into trouble with the memory usage sometimes
Ian Linden: and that is a bug
Ian Linden: which we will very much try to fix
Ian Linden: which should reduce the problems that do occur
Ian Linden: next q?

Jeska Linden: Wayfinder Wishbringer: Of similar interest... for months we have asked for multiple-manager access of estate tools in private sims. I LL recognizing this need and planning to provide such?

Ian Linden: yes, see the first question
Ian Linden: I don't think the form these tools will take are yet set in stone
Ian Linden: so we'd love to hear your wishes
Ian Linden: next q?>

Jeska Linden: Vx Shaw: Regarding performance, is there a benefit to replicating the same script, as opposed to similar scripts performing similar functions - for example, if an automated vendor was standardized within my sim?

Ian Linden: not really
Ian Linden: however, inasmuch as vendors are standardize, if you can update them in an automated way
Ian Linden: there are alot of benefits to that
Ian Linden: especially if you are able to optimize the script, etc
Ian Linden: but no implicit benefit to uniformity
Ian Linden: next?

Jeska Linden: Cutter Rubio: Right now, if we want a new height map (RAW file) applied to the sim, LL has to do it - any plans to allow estate owners to do that themselves? Would this be one of the future tool possibilities?

Ian Linden: daniel, got anything on that one?
Ian Linden: I believe that this is a step we'll *have* to take eventually
Ian Linden: manual processes just don't work as we scale out
Ian Linden: but I'm not sure if that's near or long term

Daniel Linden: We want to automated as many steps of the Estate purchase process as possible...
Daniel Linden: ....and uploading land form is also something our internal content team wants to make easier...
Daniel Linden: this point, our internal tools aren't good enough to allow what you're asking, but everyone wants to ease this part of the process!

Ian Linden: I can certainly imagine uploading a raw file to a sim and having it load it in on the fly
Ian Linden: but imagination and reality take a while to get together
Ian Linden: next?

Jeska Linden: cua Curie: What is the current timeline for SL upgrades? What is the target timeframe for adding HTML on a prim? Havok2? mono and the new rendering system?

Cyn Linden: Sure

Ian Linden: woah
Ian Linden: havok2 is one of our big projects right now
Ian Linden: since we wanted it out for 1.7
Ian Linden: it's still coming along nively
Ian Linden: nicely
Ian Linden: HTML on a prim is into the "next thing" category, so a little farther out
Ian Linden: new rendering system is still way out there, as we've slowed down on that
Ian Linden: to fix problems w/ the current version
Ian Linden: mono is anybody's guess
Ian Linden: we have a working prototype
Ian Linden: but it's going to take a while to hammer out the bugs and get the performance where we want it
Ian Linden: all of these thing, ultimately, we come out when they're done
Ian Linden: we've had poor luck giving out release dates in the past
Ian Linden: so I won't embarass myself by repeating past mistakes
Ian Linden: next?

Jeska Linden: Mulch Ennui: regarding the " memory usage" bug you just mentioned, what are the symptoms of this bug in a stacked server situation?

Ian Linden: I haven't studied it in 1.7 enough to give a good breakdown
Ian Linden: but the main symptom will be a rapid drop in performance for the same number of avs
Ian Linden: and the same general load
Ian Linden: the first thing that happens is, the autosave process (which eats extra memory)
Ian Linden: will start impacting the sim
Ian Linden: so you'll see these hourly drops in framerate which don't last very long
Ian Linden: the next thing that happens is, the autosaves from the OTHER sims start doing it
Ian Linden: so then you have multiple drops/hour, on a regular schedule
Ian Linden: the end stage is, all sims on that machine slow down because they're swapping
Ian Linden: it's not pretty, but it doesn't often get that bad
Ian Linden: I won't deny that there are memory leaks in the sim code
Ian Linden: well, here's the thing
Ian Linden: along with the legitimate cases of this occurence
Ian Linden: there have been alot of problem blamed on "stacking" that really aren't a result of stacking
Ian Linden: so it probably sounds a bit contradictory in the end
Ian Linden: since we're respoding to different complains
Ian Linden: and the day when there are no problems, I can retire
Ian Linden: I doubt it
Ian Linden: I don't think people know the best ways to cause a sim to bloat
Ian Linden: I know I don't
Ian Linden: yes it would but there are logistical problems with upgrading 900 machines
Ian Linden: and it wouldn't realy solve the problem
Ian Linden: I'd rather fix the leaks
Ian Linden: honestly if a griefer could consistently cause a sim to bloat up to a huge size quickly
Ian Linden: it would really help our debugging efforts
Ian Linden: as it is, most sims takes a long time to grow, if they do at all
Ian Linden: which is why the problems intensify which we haven't done a patch in several weeks
Ian Linden: with 1.7 there seems little risk of that for a while though
Ian Linden: next?

Jeska Linden: Fizik Baskerville: the current island layouts are a total mess, is there any plans to create a more ordered and navigatable system ?

Ian Linden: well we've certainly talked about ways to do that
Ian Linden: and I think most of that got drowned in more immediate problems like fixing the sim list in the map
Ian Linden: I don't think we arrived at anything really solid
Ian Linden: so we'd like to hear your ideas
Ian Linden: but yeah it's not an easy problem
Ian Linden: since geographic location has alot to do with identity
Ian Linden: next?

Jeska Linden: paulie Femto: How is the program for providing offshore "void" sims, attached to estates, coming along?

Ian Linden: daniel?

Cyn Linden: I can take that
Cyn Linden: We are ready to do some more beta testing with void sims
Cyn Linden: You can email me for details

Jeska Linden: Nexus Nash: what % of load does the SL software take on the servers?

Ian Linden: all of it
Ian Linden: all of the time
Ian Linden: which is why SL uses 220kW :(
Ian Linden: next?

Jeska Linden: Surreal Farber: heh.. in regards to previous question on sim location... aren't sim adresses dynamic now, so moving doesn't change your LMs?

Ian Linden: if we do it right ;)
Ian Linden: but that's not a very smooth process
Ian Linden: if it actually works at all, I'm not really up on that one
Ian Linden: because landmarks are actually still referencing a location
Ian Linden: when a sim moves we have to update them
Ian Linden: which isn't very efficient
Ian Linden: so yes/no
Ian Linden: nexT?

Jeska Linden: We've got time for one or two more questions from the queue.
Jeska Linden: Nyteshade Vesperia: Are there plans to allow people to run sims using their own hardware?

Ian Linden: not in the near future
Ian Linden: long term there are various business scenarios in which that could happen
Ian Linden: but there are a TON of issues with that
Ian Linden: not least of which is the implicit DRM [digital rights management] embodied by the permissions system
Ian Linden: which wouldn't work in that scenario
Ian Linden: paulie: I dunno, our server building has 50,000 square feet of empty space
Ian Linden: I can fit alot of sims in there
Ian Linden: next?

Jeska Linden: Anshe Chung: When you "sell" us so-called "land" or "sims", do we obtain something that maps to one certain value of computing resource that improve as technology improves and hardware price drop...
Jeska Linden: ...or did we buy into something that continuously devalues through technology improvement. Will you pass on hardware improvement to land owners? Or will our "land" be devalued? Or will you use advances to just decrease your costs and improve profit
Jeska Linden: margins.... I paid 1250$ for one sim. Does this mean that in 2 years I still own 1250$ worth of simulation power?

Ian Linden: as stated before, we will upgrade old server, so in theory if you keep a sim for 10 years
Ian Linden: you won't end up w/ a 10-year-old sim
Ian Linden: I'm not eager to comment on what our pricing will be in the future
Ian Linden: since we don't have any real plans
Ian Linden: but i'm sure things will change
Ian Linden: I like paulie's answer
Ian Linden: I think the question is, are paying less for our servers while charging residents the same amount
Ian Linden: our per-cpu purchase cost goes up and down
Ian Linden: but stayes pretty stable
Ian Linden: note that each class of sim is a bit faster than the last
Ian Linden: 5 years from now, who knows?
Ian Linden: but note that mainland sims start at $1000
Ian Linden: I would expect to continue to pay a premium for private islands
Ian Linden: we could sell for $1 if actually putting the thing online were free
Ian Linden: which it isn't right now
Ian Linden: even if the hardware were free
Ian Linden: I guess, long term, expect that we'll try to keep the prices fair
Ian Linden: we're not interested in gouging our customers
Ian Linden: it could happen
Ian Linden: that latter is probably more likely
Ian Linden: ultimately I personally think we're all better off if owning land in SL is a real investment
Ian Linden: and if the price remains stable
Ian Linden: next?

Jeska Linden: Ok, last question -- Wayfinder Wishbringer: Jeska, I think a major question that is underlying here is why did LL decide to stack servers without informing/asking the clients who pay for those servers? Wouldn't an inexpensive dual-core 1 sim/1 server ha
Jeska Linden: (thanks)

Ian Linden: well the main reason we did that was to save space and electricity
Ian Linden: well, again, we've been stacking for a long time
Ian Linden: and I think generally the results are good
Ian Linden: because no one thinks about the other case
Ian Linden: when you have 4 sims one big machine, if 2 sims are small and light, there's more free room for 2 heavy ones before there are any problems
Ian Linden: likewise they all can share their various caches
Ian Linden: and are quicker to deploy updates to
Ian Linden: so I think that stacking really is a net positive
Ian Linden: and we're quite sure that the performance is good, as long as the sims aren't too bloated
Ian Linden: and they have to get pretty darn bloated before things start slowing down
Ian Linden: so I think what we all really want is for us to fix our memory leaks
Ian Linden: I will say, BTW, that this wasn't an obvious outcome
Ian Linden: when we first tested 4 sims on 2 dual-core machines it was a disaster
Ian Linden: the performance was terrible and we never would have put that in production
Ian Linden: but we were able to resolve the issue by moving to a 64-bit kernel
Ian Linden: and enabling NUMA
Ian Linden: which brought the performance back
Ian Linden: so we did consider the results carefully
Ian Linden: ok, I guess that's all the time we have but I can hang around here for a few more minutes..
Ian Linden: I think that's the "void" sim question from earlier

Daniel Linden: Thanks Ian!

Cyn Linden: Thanks for coming everyone

Jeska Linden: Thanks for the good questions ;)

Cyn Linden: Thank you Ian and Jes!

Ian Linden: I think probably there's been some internal confusion about the nature of memory leaks
Ian Linden: and again, we want to avoid creating a sim bloat boogey-man that gets blamed for everything
Ian Linden: I will post some server specs on the sim stats wiki
Ian Linden: but believe me, you're better off sharing a class 4 machine with 3 other sims
Ian Linden: than alone on a class 2 machine
Ian Linden: Mulch: I think the jury's still out on that one
Ian Linden: cua: there's clearly a need for better script messaging and data storage
Ian Linden: cua: lots of stuff in the works there, but I won't want to announce it yet because the specifics are still in flux
Ian Linden: cua: well that's a new 1.7 bug I'm sure
Ian Linden: cua: we'll look into that
Ian Linden: Mulch: I'd love to hear what the new scheduling has done to your scripts?
Ian Linden: Mulch: I they just slower, or is it something else?
Ian Linden: wayfinder: no clue
Ian Linden: what I'd really like is a copy of a script that broke w/ 1.7
Ian Linden: or an object
Ian Linden: which I can give to QA
Ian Linden: alright, I'll refer the bug hunters to racerx
Ian Linden: a bug w/ a the havok interface is certinaly possible
Ian Linden: ok
Ian Linden: anything else?
Ian Linden: I though this was going to be hard

Cyn Linden: OK guys!

Ian Linden: thomson, check

Cyn Linden: I dunno about you but I have telehubs to set :)
Cyn Linden: :)
Cyn Linden: Thanks again everyone for coming
Cyn Linden: Anytime

Ian Linden: k, thanks folks, I'll have some sim class stuff in the wiki page I sent out earlier up in a little while

Cyn Linden: Haven't decided yet
Cyn Linden: I will keep you informed, and you can always IM me

Ian Linden: wayfinder: mostly better higher
Ian Linden: but we put some class 3s in the old class 1 slots
Ian Linden: just to confuse everybody
Ian Linden: well, IP addresses are a limited resource
Ian Linden: but I agree we could have done it better
Ian Linden: it didn't seem relevent at the time
Ian Linden: and it won't be if we ever get the class info into the about box
Ian Linden: Mike: maybe
Ian Linden: sure
Ian Linden: it could be in theory
Ian Linden: there is only one
Ian Linden: wayfinder: BUT
Ian Linden: hah, still plain old pata
Ian Linden: but the newer disks are quite a bit faster than the old ones
Ian Linden: and, more importantely, the sims really aren't I/O bound
Ian Linden: for one thing, the IO is threaded
Ian Linden: so IO won't block the server or slow it down
Ian Linden: it just means that, if it is being slow, assets etc will load more slowly
Ian Linden: but we don't see that much, even on the class 4 machines
Ian Linden: Wayfinder: basically it means that all the reads/writes happen out of step with the main process
Ian Linden: wayfinder, so you're not waiting for it the way you do with most desktop apps
Ian Linden: sortof
Ian Linden: the real point is, class 5 will probably not have extra-fast disks
Ian Linden: because there doesn't appear to be a need
Ian Linden: whatever the next class is ;)
Ian Linden: it's probbaly a year out
Ian Linden: well, the asset servers are a different story altogether
Ian Linden: THEY have the higher-end SATA disks, and LOTS of them
Ian Linden: doubt it
Ian Linden: more likely, if we want to provide access to that sort of data, it'll come in a web service
Ian Linden: actually, the asset servers are using a very weird proprietary filesystem
Ian Linden: that does striping in a way that isn't the same as raid
Ian Linden: sortof
Ian Linden: apache on top of the clustered filesystem
Ian Linden: wayfinder: yes, a way to profile your script load and deputized control are important
Ian Linden: but remember we have to balance that against, say, the lots and lots of people who still have frequent crashes
Ian Linden: right
Ian Linden: don't think we're not aware of the problem there
Ian Linden: it's frustrating that we can't address this stuff more quickly
Ian Linden: we've been planning for a long time
Ian Linden: but, you know, people voted, and havok 2 was at the top
Ian Linden: so that's what we're working on
Ian Linden: yeah
Ian Linden: it's supported alright
Ian Linden: havok 3 is mostly just havok 2 w/ ragdolls
Ian Linden: I didn't think so, but I won't argue w/ it
Ian Linden: I'm not heavily into the havok stuff
Ian Linden: all I know is that the decision was to skip it for now
Ian Linden: ah
Ian Linden: well then I just don' tknow
Ian Linden: ah, region crossing

Jeska Linden: Ok, thanks again for coming out everyone.

Ian Linden: region crossing is always a release away from being fixed
Ian Linden: someday we'll get it right

Cyn Linden: :D

Jeska Linden: Bye everyone!

Ian Linden: k, thanks all, I'm off

Cyn Linden: See ya guys!

Ad blocker interference detected!

Wikia is a free-to-use site that makes money from advertising. We have a modified experience for viewers using ad blockers

Wikia is not accessible if you’ve made further modifications. Remove the custom ad blocker rule(s) and the page will load as expected.